[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:28 dist-test-slave-3q8k krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:28 dist-test-slave-3q8k krb5kdc[399](info): set up 2 sockets
Sep 24 00:09:28 dist-test-slave-3q8k krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 00:09:29.291795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:29.414502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:29.500468 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:29.500785 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:29.500813 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:29.502130 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:29.502187 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:29.502202 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:29.502216 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:29.502228 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:29.503870 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:29.504173 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:29.504453 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:29.509891 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:29.510027 412 server_base.cc:1034] running on GCE node
W20260924 00:09:29.509912 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:29.509860 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:29.510414 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:29.510715 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:29.511888 412 hybrid_clock.cc:648] HybridClock initialized: now 1790208569511879 us; error 46 us; skew 500 ppm
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:29.514663 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:29.515153 412 webserver.cc:460] Webserver started at http://127.0.98.62:41297/ using document root <none> and password file <none>
I20260924 00:09:29.515424 412 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:29.515502 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:29.515671 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:29.516762 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "a5a75ee195d04e7d9ee8818adccf54f6"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "1d1e9cf12d48a1830efae361525b7447"
server_key_iv: "265cdd9463fa9157301ffc47746150b1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.517156 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5a75ee195d04e7d9ee8818adccf54f6"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "1d1e9cf12d48a1830efae361525b7447"
server_key_iv: "265cdd9463fa9157301ffc47746150b1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.518637 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:29.519577 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.519763 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:29.519876 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "a5a75ee195d04e7d9ee8818adccf54f6"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "1d1e9cf12d48a1830efae361525b7447"
server_key_iv: "265cdd9463fa9157301ffc47746150b1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.519953 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:29.530781 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:29.533118 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:29.533319 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:29.537719 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44647
I20260924 00:09:29.537806 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44647 every 8 connection(s)
I20260924 00:09:29.538239 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:29.539151 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:29.539696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 412
I20260924 00:09:29.539808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:29.540606 392 external_mini_cluster.cc:1428] Setting key 3734b6db07628ba924d0c94b78715e6d
I20260924 00:09:29.542609 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6: Bootstrap starting.
I20260924 00:09:29.543403 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:29.543733 480 log.cc:824] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:29.544515 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6: No bootstrap required, opened a new log
I20260924 00:09:29.546648 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260924 00:09:29.546825 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:29.546862 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a75ee195d04e7d9ee8818adccf54f6, State: Initialized, Role: FOLLOWER
I20260924 00:09:29.547001 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260924 00:09:29.547087 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:29.547137 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:29.547194 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:29.547984 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260924 00:09:29.548110 480 leader_election.cc:302] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a75ee195d04e7d9ee8818adccf54f6; no voters:
I20260924 00:09:29.548280 480 leader_election.cc:288] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:29.548334 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:29.548486 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [term 1 LEADER]: Becoming Leader. State: Replica: a5a75ee195d04e7d9ee8818adccf54f6, State: Running, Role: LEADER
I20260924 00:09:29.548537 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:29.548583 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260924 00:09:29.548866 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a75ee195d04e7d9ee8818adccf54f6. Latest consensus state: current_term: 1 leader_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } } }
I20260924 00:09:29.548955 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:29.548877 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a75ee195d04e7d9ee8818adccf54f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } } }
I20260924 00:09:29.549091 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:29.549218 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:29.549715 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:29.551126 502 mvcc.cc:205] Tried to move back new op lower bound from 7332694300879949824 to 7332694300871393280. Current Snapshot: MvccSnapshot[applied={T|T < 7332694300879949824}]
I20260924 00:09:29.551518 491 catalog_manager.cc:1347] Generated new cluster ID: d4f28be7aa634ff78ad2dc4257d12404
I20260924 00:09:29.551571 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:29.558707 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:29.541578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.98.62:44647)
0924 00:09:29.541768 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:29.541777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:29.542101 (+ 324us) server_negotiation.cc:407] Connection header received
0924 00:09:29.544908 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:29.544915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:29.544960 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:29.545068 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:29.547210 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.549008 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.550587 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.551158 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.551291 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:29.554912 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:29.554916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:29.554916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:29.554928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:29.556768 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.557239 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.557243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.557243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.557339 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.557617 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.557618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.557618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.557857 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:29.558024 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:29.558309 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 00:09:29.558448 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260924 00:09:29.566843 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:29.567610 491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:29.575378 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a75ee195d04e7d9ee8818adccf54f6: Generated new TSK 0
I20260924 00:09:29.575594 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:29.605724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:29.705183 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:29.705396 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:29.705421 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:29.706614 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:29.706660 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:29.706688 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:29.708181 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:29.708431 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:29.708689 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:29.713805 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:29.713843 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:29.713805 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:29.714114 506 server_base.cc:1034] running on GCE node
I20260924 00:09:29.714306 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:29.714553 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:29.715710 506 hybrid_clock.cc:648] HybridClock initialized: now 1790208569715698 us; error 43 us; skew 500 ppm
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:29.718281 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:29.718750 506 webserver.cc:460] Webserver started at http://127.0.98.1:43169/ using document root <none> and password file <none>
I20260924 00:09:29.718981 506 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:29.719056 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:29.719208 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:29.720305 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "915e1a53416f427eaad1df5f97c853d1"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "44233c6fd741be59f165979a43b12a33"
server_key_iv: "b61e946266a004305f65e4c4f8ff56a1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.720674 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "915e1a53416f427eaad1df5f97c853d1"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "44233c6fd741be59f165979a43b12a33"
server_key_iv: "b61e946266a004305f65e4c4f8ff56a1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.722055 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:29.722841 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.723035 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:29.723107 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "915e1a53416f427eaad1df5f97c853d1"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "44233c6fd741be59f165979a43b12a33"
server_key_iv: "b61e946266a004305f65e4c4f8ff56a1"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.723157 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:29.731123 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:29.732973 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:29.733089 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:29.733286 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:29.733552 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:29.733584 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.733606 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:29.733618 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.739624 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36681
I20260924 00:09:29.739665 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36681 every 8 connection(s)
I20260924 00:09:29.739980 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:29.741214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 506
I20260924 00:09:29.741284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:29.741451 392 external_mini_cluster.cc:1428] Setting key 6e091645fd6b9473db4fbdb0699b0019
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:09:29.750442 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:29.740652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52655 (local address 127.0.98.62:44647)
0924 00:09:29.740716 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:29.740718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:29.740886 (+ 168us) server_negotiation.cc:407] Connection header received
0924 00:09:29.741458 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:29.741459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:29.741471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:29.741568 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:29.742649 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.743296 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.744451 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.744653 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.744710 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:29.747188 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:29.747190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:29.747191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:29.747199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:29.748877 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.749414 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.749416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.749416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.749476 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.749772 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.749773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.749773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.749929 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:29.750006 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:29.750307 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 00:09:29.750375 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:09:29.750619 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:29.740596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.1:52655)
0924 00:09:29.740760 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:29.740801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:29.741337 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:29.741685 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:29.741690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:29.742345 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:29.742541 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:29.742546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.743383 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.743384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:29.744361 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:29.744363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.744741 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.744742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:29.744889 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:29.745495 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:29.745506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:29.747090 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:29.748972 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:29.748979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:29.748980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:29.749318 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:29.749582 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:29.749583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:29.749583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:29.749705 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:29.749997 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:29.750002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:29.750180 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 00:09:29.750270 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:09:29.750391 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260924 00:09:29.751147 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260924 00:09:29.751281 635 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:29.751443 635 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260924 00:09:29.752007 444 ts_manager.cc:194] Registered new tserver with Master: 915e1a53416f427eaad1df5f97c853d1 (127.0.98.1:36681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 00:09:29.752775 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:29.766431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:29.848826 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:29.849064 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:29.849092 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:29.850329 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:29.850406 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:29.850454 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:29.851889 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:29.852085 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:29.852264 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:29.857518 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:29.857491 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:29.857499 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:29.857749 640 server_base.cc:1034] running on GCE node
I20260924 00:09:29.857954 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:29.858250 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:29.859416 640 hybrid_clock.cc:648] HybridClock initialized: now 1790208569859395 us; error 37 us; skew 500 ppm
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:29.861994 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:29.862528 640 webserver.cc:460] Webserver started at http://127.0.98.2:46269/ using document root <none> and password file <none>
I20260924 00:09:29.862769 640 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:29.862840 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:29.862977 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:29.864163 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "059fdd9727b34d119f2ed38139046d7b"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "e0ae6c1cd3546f064d99358f0ca51e4f"
server_key_iv: "e98fd035640a6ebebe56aee031bec663"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.864549 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "059fdd9727b34d119f2ed38139046d7b"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "e0ae6c1cd3546f064d99358f0ca51e4f"
server_key_iv: "e98fd035640a6ebebe56aee031bec663"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.865949 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:09:29.866837 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.867183 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:29.867321 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "059fdd9727b34d119f2ed38139046d7b"
format_stamp: "Formatted at 2026-09-24 00:09:29 on dist-test-slave-3q8k"
server_key: "e0ae6c1cd3546f064d99358f0ca51e4f"
server_key_iv: "e98fd035640a6ebebe56aee031bec663"
server_key_version: "encryptionkey@0"
I20260924 00:09:29.867410 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:29.874279 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:29.876627 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:29.876794 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:29.877058 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:29.877369 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:29.877430 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.877480 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:29.877511 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:29.883988 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43337
I20260924 00:09:29.884038 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43337 every 8 connection(s)
I20260924 00:09:29.884392 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:09:29 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:29.891198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 640
I20260924 00:09:29.891364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:29.891578 392 external_mini_cluster.cc:1428] Setting key ca844636f97e452c67b31fa5268f3465
I20260924 00:09:29.894780 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:29.885245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49757 (local address 127.0.98.62:44647)
0924 00:09:29.885333 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:29.885336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:29.885447 (+ 111us) server_negotiation.cc:407] Connection header received
0924 00:09:29.885938 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:29.885939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:29.885949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:29.886018 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:29.887041 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.887650 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.888725 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.888939 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.889015 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:29.891557 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:29.891558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:29.891559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:29.891574 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:29.893148 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.893700 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.893701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.893701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.893756 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:29.894058 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:29.894060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:29.894060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:29.894180 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:29.894293 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:29.894582 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 00:09:29.894686 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:09:29.895005 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:29.885149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.2:49757)
0924 00:09:29.885318 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:29.885355 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:29.885844 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:29.886108 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:29.886111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:29.886718 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:29.886924 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:29.886929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.887741 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.887742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:29.888608 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:29.888610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:29.889098 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:29.889099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:29.889221 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:29.889826 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:29.889834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:29.891435 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:29.893293 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:29.893300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:29.893302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:29.893625 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:29.893822 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:29.893823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:29.893823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:29.893965 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:29.894284 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:29.894291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:29.894438 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 00:09:29.894559 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:09:29.894729 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260924 00:09:29.895530 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260924 00:09:29.895638 769 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:29.895854 769 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260924 00:09:29.896243 444 ts_manager.cc:194] Registered new tserver with Master: 059fdd9727b34d119f2ed38139046d7b (127.0.98.2:43337)
I20260924 00:09:29.896632 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49757
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:29.919270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:30.014605 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:30.014819 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:30.014843 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:30.016076 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:30.016125 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:30.016153 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:30.017608 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:30.017863 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:30.018087 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:30.023142 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:30.023134 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:30.023108 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:30.023313 774 server_base.cc:1034] running on GCE node
I20260924 00:09:30.023545 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:30.023825 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:30.024966 774 hybrid_clock.cc:648] HybridClock initialized: now 1790208570024952 us; error 29 us; skew 500 ppm
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:30.027498 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:30.027966 774 webserver.cc:460] Webserver started at http://127.0.98.3:33093/ using document root <none> and password file <none>
I20260924 00:09:30.028196 774 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:30.028242 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:30.028378 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:30.029711 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "02dc064d92d7432f87d667284e942324"
format_stamp: "Formatted at 2026-09-24 00:09:30 on dist-test-slave-3q8k"
server_key: "48b7d3df07e3dff3e4a523176e0a49e0"
server_key_iv: "8c329b9e8b394071fd96d7e24212513f"
server_key_version: "encryptionkey@0"
I20260924 00:09:30.030094 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02dc064d92d7432f87d667284e942324"
format_stamp: "Formatted at 2026-09-24 00:09:30 on dist-test-slave-3q8k"
server_key: "48b7d3df07e3dff3e4a523176e0a49e0"
server_key_iv: "8c329b9e8b394071fd96d7e24212513f"
server_key_version: "encryptionkey@0"
I20260924 00:09:30.031495 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:09:30.032302 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:30.032454 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:30.032541 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "02dc064d92d7432f87d667284e942324"
format_stamp: "Formatted at 2026-09-24 00:09:30 on dist-test-slave-3q8k"
server_key: "48b7d3df07e3dff3e4a523176e0a49e0"
server_key_iv: "8c329b9e8b394071fd96d7e24212513f"
server_key_version: "encryptionkey@0"
I20260924 00:09:30.032596 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:30.054741 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:30.057169 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:30.057301 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:30.057493 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:30.057853 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:30.057895 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:30.057919 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:30.057969 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:30.064535 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43489
I20260924 00:09:30.064628 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43489 every 8 connection(s)
I20260924 00:09:30.064924 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:30.074298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 774
I20260924 00:09:30.074412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:30.074515 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:30.065691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43269 (local address 127.0.98.62:44647)
0924 00:09:30.065757 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:30.065759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:30.065940 (+ 181us) server_negotiation.cc:407] Connection header received
0924 00:09:30.066479 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:30.066480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:30.066494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:30.066538 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:30.067554 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.068056 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.069075 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.069273 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.069384 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:30.071719 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:30.071721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:30.071721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:30.071729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:30.073111 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.073544 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.073546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.073546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.073608 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.073861 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.073862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.073862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.073977 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:30.074054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:30.074344 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 00:09:30.074422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:09:30.074647 392 external_mini_cluster.cc:1428] Setting key 629df9f52dc9f5d9ce8f093d442063ca
I20260924 00:09:30.074837 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:30.065668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.3:43269)
0924 00:09:30.065821 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:30.065855 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:30.066371 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:30.066608 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:30.066616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:30.067193 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:30.067459 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:30.067463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.068172 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.068173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:30.068969 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:30.068971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.069393 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.069393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:30.069522 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:30.070024 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:30.070032 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:30.071605 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:30.073236 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:30.073243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:30.073246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:30.073465 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:30.073677 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:30.073678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:30.073679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:30.073786 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:30.074043 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:30.074049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:30.074208 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 00:09:30.074313 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 00:09:30.074472 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260924 00:09:30.075383 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260924 00:09:30.075517 903 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:30.075711 903 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260924 00:09:30.076118 443 ts_manager.cc:194] Registered new tserver with Master: 02dc064d92d7432f87d667284e942324 (127.0.98.3:43489)
I20260924 00:09:30.076547 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43269
I20260924 00:09:30.086990 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:30.111011 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:30.103198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.98.1:36681)
0924 00:09:30.103401 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:30.103406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:30.103417 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:30.103444 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:30.103446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:30.103461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:30.103529 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:30.104126 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.104851 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.105609 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.105911 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.105967 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:30.108391 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:30.108399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:30.108400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:30.108406 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:30.109854 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.110189 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.110192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.110192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.110258 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.110526 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.110527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.110527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.110676 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:30.110813 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:30.110859 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 00:09:30.110923 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
W20260924 00:09:30.111495 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57518
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:30.133621 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:30.126289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.98.1:36681)
0924 00:09:30.126356 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:30.126358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:30.126443 (+ 85us) server_negotiation.cc:407] Connection header received
0924 00:09:30.126505 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:30.126505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:30.126518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:30.126568 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:30.127177 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.127727 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.128514 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:30.128737 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:30.128801 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:30.131181 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:30.131182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:30.131182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:30.131190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:30.132589 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.132926 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.132927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.132927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.132979 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:30.133190 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:30.133190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:30.133190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:30.133282 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:30.133373 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:30.133455 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 00:09:30.133517 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:09:30.134256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 506
I20260924 00:09:30.139062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 640
I20260924 00:09:30.144992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 774
I20260924 00:09:30.149408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 412
2026-09-24T00:09:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1961 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[917](info): set up 2 sockets
Sep 24 00:09:30 dist-test-slave-3q8k krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 00:09:31.194769 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:31.314694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:31.394274 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:31.394492 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:31.394515 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:31.395764 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:31.395815 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:31.395829 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:31.395843 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:31.395854 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:31.397360 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:31.397610 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:31.397857 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:31.403077 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.403170 930 server_base.cc:1034] running on GCE node
W20260924 00:09:31.403234 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.403112 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.403548 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:31.403846 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:31.405081 930 hybrid_clock.cc:648] HybridClock initialized: now 1790208571405060 us; error 51 us; skew 500 ppm
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:31.407653 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:31.408172 930 webserver.cc:460] Webserver started at http://127.0.98.62:43919/ using document root <none> and password file <none>
I20260924 00:09:31.408402 930 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:31.408470 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:31.408622 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:31.409700 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "b517668a5b1c49f18ee8ce46df04b7c1"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "dfe58677d94876a86f20fe9231041b82"
server_key_iv: "94d01c91e2ce9ffee946d74a888156ae"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.410069 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "b517668a5b1c49f18ee8ce46df04b7c1"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "dfe58677d94876a86f20fe9231041b82"
server_key_iv: "94d01c91e2ce9ffee946d74a888156ae"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.411477 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:09:31.412250 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.412418 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:31.412520 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "b517668a5b1c49f18ee8ce46df04b7c1"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "dfe58677d94876a86f20fe9231041b82"
server_key_iv: "94d01c91e2ce9ffee946d74a888156ae"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.412593 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:31.423998 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:31.425818 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:31.425988 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:31.429912 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33825
I20260924 00:09:31.429961 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33825 every 8 connection(s)
I20260924 00:09:31.430337 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:31.431371 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:31.433833 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: Bootstrap starting.
I20260924 00:09:31.434518 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:31.434844 998 log.cc:824] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:31.435813 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: No bootstrap required, opened a new log
I20260924 00:09:31.437582 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260924 00:09:31.437732 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:31.437757 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b517668a5b1c49f18ee8ce46df04b7c1, State: Initialized, Role: FOLLOWER
I20260924 00:09:31.437949 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260924 00:09:31.438022 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:31.438046 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:31.438076 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:31.438673 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260924 00:09:31.438793 998 leader_election.cc:302] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b517668a5b1c49f18ee8ce46df04b7c1; no voters:
I20260924 00:09:31.438997 998 leader_election.cc:288] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:31.439080 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:31.439306 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:31.439368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 930
I20260924 00:09:31.439446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:31.439607 392 external_mini_cluster.cc:1428] Setting key f5cfac5df3625c82450ad4b81b2e31a8
I20260924 00:09:31.440045 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [term 1 LEADER]: Becoming Leader. State: Replica: b517668a5b1c49f18ee8ce46df04b7c1, State: Running, Role: LEADER
I20260924 00:09:31.440140 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } }
I20260924 00:09:31.440487 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b517668a5b1c49f18ee8ce46df04b7c1. Latest consensus state: current_term: 1 leader_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } } }
I20260924 00:09:31.440569 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:31.441182 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b517668a5b1c49f18ee8ce46df04b7c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33825 } } }
I20260924 00:09:31.441428 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1 [sys.catalog]: This master's current role is: LEADER
W20260924 00:09:31.441602 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:09:31.441649 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:09:31.441727 1018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:31.441907 1018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:31.443580 1018 catalog_manager.cc:1347] Generated new cluster ID: e22988f877a648b4ba44db42de9077ce
I20260924 00:09:31.443629 1018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:31.452132 1018 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:31.452332 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.440105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.0.98.62:33825)
0924 00:09:31.440846 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.440859 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.440894 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:09:31.441351 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.441362 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.441435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.441619 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:31.442378 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.443753 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.444848 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.445469 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.445587 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.448432 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:31.448436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:31.448437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:31.448449 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:31.450377 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.450778 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.450781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.450782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.450884 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.451106 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.451107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.451108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.451420 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:31.451573 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.451919 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.452096 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":60,"threads_started":1}
I20260924 00:09:31.452752 1018 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:31.457101 1018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b517668a5b1c49f18ee8ce46df04b7c1: Generated new TSK 0
I20260924 00:09:31.457280 1018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:31.487437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:31.584153 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:31.584353 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:31.584378 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:31.585542 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:31.585584 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:31.585611 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:31.587020 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:31.587294 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:31.587543 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:31.592474 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.592485 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.592484 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.593051 1024 server_base.cc:1034] running on GCE node
I20260924 00:09:31.593297 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:31.593540 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:31.594709 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790208571594696 us; error 29 us; skew 500 ppm
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:31.597421 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:31.597898 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45921/ using document root <none> and password file <none>
I20260924 00:09:31.598129 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:31.598198 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:31.598352 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:31.599519 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae14adb618e143f0a034576cf0ab2ee7"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "d6b88b7fef9ccf63db99132b0cbbea85"
server_key_iv: "47b6cbbdbd2f73fbe6db09acc6287e48"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.599877 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae14adb618e143f0a034576cf0ab2ee7"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "d6b88b7fef9ccf63db99132b0cbbea85"
server_key_iv: "47b6cbbdbd2f73fbe6db09acc6287e48"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.601243 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:31.602020 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.602180 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:31.602286 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "ae14adb618e143f0a034576cf0ab2ee7"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "d6b88b7fef9ccf63db99132b0cbbea85"
server_key_iv: "47b6cbbdbd2f73fbe6db09acc6287e48"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.602371 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:31.629864 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:31.631860 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:31.631969 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:31.632150 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:31.632431 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:31.632460 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.632483 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:31.632495 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.638321 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39011
I20260924 00:09:31.638403 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39011 every 8 connection(s)
I20260924 00:09:31.638729 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:31.642976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1024
I20260924 00:09:31.643066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:31.643273 392 external_mini_cluster.cc:1428] Setting key fc92a155c5b6e549f1b339012691c0af
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:31.649086 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.639513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34827 (local address 127.0.98.62:33825)
0924 00:09:31.639574 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.639576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.639847 (+ 271us) server_negotiation.cc:407] Connection header received
0924 00:09:31.640450 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.640452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.640470 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.640539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:31.641548 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.642089 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.643111 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.643443 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.643528 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.646240 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:31.646242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:31.646242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:31.646249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:31.647732 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.648135 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.648136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.648137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.648200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.648457 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.648458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.648458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.648569 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:31.648635 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.648911 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.648991 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:09:31.649313 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.639545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33825 (local address 127.0.98.1:34827)
0924 00:09:31.639691 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.639735 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.640352 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.640595 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.640598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.641243 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:31.641431 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.641436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.642199 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.642200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.642989 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.642991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.643609 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.643610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.643744 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.644256 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:31.644266 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:31.646128 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:31.647821 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.647828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.647830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.648050 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.648264 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.648265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.648265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.648379 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.648639 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:31.648653 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:31.648785 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.648891 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.649065 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":88,"threads_started":1}
I20260924 00:09:31.649847 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33825
I20260924 00:09:31.649950 1153 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:31.650121 1153 heartbeater.cc:511] Master 127.0.98.62:33825 requested a full tablet report, sending...
I20260924 00:09:31.650686 962 ts_manager.cc:194] Registered new tserver with Master: ae14adb618e143f0a034576cf0ab2ee7 (127.0.98.1:39011)
I20260924 00:09:31.651360 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34827
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:31.670344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:31.751565 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:31.751780 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:31.751806 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:31.752954 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:31.752991 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:31.753018 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:31.754393 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:31.754587 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:31.754773 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:31.759927 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.759959 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.759970 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.759986 1158 server_base.cc:1034] running on GCE node
I20260924 00:09:31.760299 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:31.760533 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:31.761682 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790208571761663 us; error 34 us; skew 500 ppm
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:31.764375 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:31.764845 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40779/ using document root <none> and password file <none>
I20260924 00:09:31.765051 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:31.765094 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:31.765185 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:31.766187 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1c58901b450494e92d4c2d5614908b3"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "8a802eefac71ecb10e45a20c82eaf8e7"
server_key_iv: "a32447f23bdd0514cafb29931b3caac5"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.766528 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1c58901b450494e92d4c2d5614908b3"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "8a802eefac71ecb10e45a20c82eaf8e7"
server_key_iv: "a32447f23bdd0514cafb29931b3caac5"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.767939 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:09:31.768632 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.768808 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:31.768911 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "e1c58901b450494e92d4c2d5614908b3"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "8a802eefac71ecb10e45a20c82eaf8e7"
server_key_iv: "a32447f23bdd0514cafb29931b3caac5"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.768986 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:31.786284 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:31.788857 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:31.789038 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:31.789288 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:31.789608 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:31.789671 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.789716 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:31.789744 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.795805 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43383
I20260924 00:09:31.795850 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43383 every 8 connection(s)
I20260924 00:09:31.796176 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:31.805318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1158
I20260924 00:09:31.805420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:31.805596 392 external_mini_cluster.cc:1428] Setting key a0aa04c5865bc69b246f8826a8c0d2cd
I20260924 00:09:31.806341 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.796892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34519 (local address 127.0.98.62:33825)
0924 00:09:31.796953 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.796955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.797226 (+ 271us) server_negotiation.cc:407] Connection header received
0924 00:09:31.797716 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.797718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.797729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.797795 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:31.798831 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.799513 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.800598 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.800784 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.800842 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.803098 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:31.803099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:31.803099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:31.803107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:31.804609 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.805117 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.805119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.805119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.805172 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.805524 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.805525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.805525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.805656 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:31.805738 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.806129 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.806217 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:09:31.806633 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.796851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33825 (local address 127.0.98.2:34519)
0924 00:09:31.797017 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.797140 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.797603 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.797877 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.797881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.798503 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:31.798708 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.798713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.799642 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.799643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.800503 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.800506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.800859 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.800859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.800958 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.801459 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:31.801467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:31.802971 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:31.804688 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.804694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.804696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.804999 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.805237 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.805238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.805239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.805401 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.805746 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:31.805753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:31.805945 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.806081 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.806300 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 00:09:31.807289 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33825
I20260924 00:09:31.807410 1287 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:31.807600 1287 heartbeater.cc:511] Master 127.0.98.62:33825 requested a full tablet report, sending...
I20260924 00:09:31.808050 962 ts_manager.cc:194] Registered new tserver with Master: e1c58901b450494e92d4c2d5614908b3 (127.0.98.2:43383)
I20260924 00:09:31.808434 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34519
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:31.830487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:31.912624 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:31.912864 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:31.912904 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:31.914098 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:31.914170 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:31.914216 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:31.915860 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:31.916148 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:31.916376 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:31.921546 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:31.921576 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.921659 1292 server_base.cc:1034] running on GCE node
W20260924 00:09:31.921581 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:31.921983 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:31.922189 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:31.923278 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790208571923270 us; error 47 us; skew 500 ppm
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:31.925859 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:31.926308 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35787/ using document root <none> and password file <none>
I20260924 00:09:31.926530 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:31.926599 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:31.926748 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:31.927889 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "e8a302294174485e831f0c4debdd3de2"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "0311b488e8fd76d37f9730631f7cf268"
server_key_iv: "408b89119935d756b56b944aa514ad65"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.928278 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a302294174485e831f0c4debdd3de2"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "0311b488e8fd76d37f9730631f7cf268"
server_key_iv: "408b89119935d756b56b944aa514ad65"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.929626 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:31.930454 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.930682 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:31.930788 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "e8a302294174485e831f0c4debdd3de2"
format_stamp: "Formatted at 2026-09-24 00:09:31 on dist-test-slave-3q8k"
server_key: "0311b488e8fd76d37f9730631f7cf268"
server_key_iv: "408b89119935d756b56b944aa514ad65"
server_key_version: "encryptionkey@0"
I20260924 00:09:31.930871 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:31.947654 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:31.950006 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:31.950171 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:31.950405 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:31.950718 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:31.950773 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.950815 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:31.950842 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:31.957181 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43501
I20260924 00:09:31.957250 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43501 every 8 connection(s)
I20260924 00:09:31.957568 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:09:31 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:31.965971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1292
I20260924 00:09:31.966110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:31.966313 392 external_mini_cluster.cc:1428] Setting key 293b9ea2c2d75cf955bd1a493556d842
I20260924 00:09:31.967861 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.958311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51491 (local address 127.0.98.62:33825)
0924 00:09:31.958381 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.958382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.958593 (+ 211us) server_negotiation.cc:407] Connection header received
0924 00:09:31.959119 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.959120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.959132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.959195 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:31.960323 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.960879 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.962020 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.962209 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.962268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.964611 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:31.964612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:31.964613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:31.964619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:31.966121 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.966651 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.966652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.966653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.966725 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.967164 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.967165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.967165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.967346 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:31.967431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.967710 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.967783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:09:31.968117 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.958293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33825 (local address 127.0.98.3:51491)
0924 00:09:31.958436 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.958477 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.958941 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.959377 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.959381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.959989 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:31.960205 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.960209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.960972 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.960973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.961935 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.961937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.962313 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.962314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.962427 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.963002 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:31.963011 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:31.964513 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:31.966245 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.966251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.966254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.966598 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.966848 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:31.966849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:31.966849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:31.967028 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:31.967405 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:31.967412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:31.967579 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.967676 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.967812 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 00:09:31.968655 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33825
I20260924 00:09:31.968781 1421 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:31.968984 1421 heartbeater.cc:511] Master 127.0.98.62:33825 requested a full tablet report, sending...
I20260924 00:09:31.969404 962 ts_manager.cc:194] Registered new tserver with Master: e8a302294174485e831f0c4debdd3de2 (127.0.98.3:43501)
I20260924 00:09:31.969856 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51491
I20260924 00:09:31.978390 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:09:31.986116 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.979956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.0.98.62:33825)
0924 00:09:31.980027 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.980029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.980110 (+ 81us) server_negotiation.cc:407] Connection header received
0924 00:09:31.980150 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.980151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.980159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.980203 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:31.980696 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.981226 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.982212 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.982390 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.982443 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.983668 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:31.983670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:31.983670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:31.983676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:31.985087 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.985402 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.985403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.985403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.985451 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:31.985618 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:31.985619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:31.985619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:31.985766 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:31.985838 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.985992 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.986050 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:09:31.988461 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:31.990345 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:31.997280 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.992884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43383 (local address 127.0.0.1:54032)
0924 00:09:31.993187 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.993262 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.993334 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.993576 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.993580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.993582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:31.993655 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.993657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.995117 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.995118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.996263 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.996265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.996944 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.996944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.997030 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.997043 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.997136 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.997186 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":41,"thread_start_us":60,"threads_started":1}
I20260924 00:09:31.997357 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.993162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.98.3:43501)
0924 00:09:31.993371 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.993375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.993545 (+ 170us) server_negotiation.cc:407] Connection header received
0924 00:09:31.993586 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.993588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.993604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.993687 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:31.994018 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.994727 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.996325 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.997044 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.997109 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.997156 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.997243 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.997279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260924 00:09:31.997439 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.993149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.98.1:39011)
0924 00:09:31.993319 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.993332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.993635 (+ 303us) server_negotiation.cc:407] Connection header received
0924 00:09:31.993852 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.993854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.993867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.993936 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:31.994047 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.995095 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.996227 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.996957 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.997029 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.997087 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.997318 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.997355 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 00:09:31.997788 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.992979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.0.98.2:43383)
0924 00:09:31.993350 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:31.993355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:31.993366 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:31.993441 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:31.993444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:31.993462 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:31.993545 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:31.993745 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.995006 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.996352 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.996837 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.996937 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.996999 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:31.997700 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 00:09:31.997736 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":107,"threads_started":1}
I20260924 00:09:31.997402 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.992892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39011 (local address 127.0.0.1:48200)
0924 00:09:31.993580 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.993589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.993634 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.993942 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.993943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.993945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:31.993984 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.993986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.995172 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.995173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.996138 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.996141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.997123 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.997124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.997188 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.997192 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.997241 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.997279 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":8,"thread_start_us":89,"threads_started":1}
I20260924 00:09:31.997480 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:31.992892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43501 (local address 127.0.0.1:59732)
0924 00:09:31.993458 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:31.993467 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:31.993516 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:31.993696 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:31.993696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:31.993698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:31.993753 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.993755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.994841 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.994842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.996244 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:31.996247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:31.997118 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:31.997119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:31.997176 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:31.997179 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:31.997231 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:09:31.997261 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":43,"thread_start_us":101,"threads_started":1}
I20260924 00:09:31.998579 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1e72cde51fea4417bfce147f20fea722 (DEFAULT_TABLE table=test-table [id=36569904e683487397cb25897229f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:31.998628 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1e72cde51fea4417bfce147f20fea722 (DEFAULT_TABLE table=test-table [id=36569904e683487397cb25897229f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:31.998579 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 1e72cde51fea4417bfce147f20fea722 (DEFAULT_TABLE table=test-table [id=36569904e683487397cb25897229f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:31.998932 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e72cde51fea4417bfce147f20fea722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:31.998932 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e72cde51fea4417bfce147f20fea722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:31.998941 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e72cde51fea4417bfce147f20fea722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:32.001464 1438 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Bootstrap starting.
I20260924 00:09:32.002084 1439 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Bootstrap starting.
I20260924 00:09:32.002810 1438 tablet_bootstrap.cc:654] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:32.003206 1439 tablet_bootstrap.cc:654] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:32.003420 1440 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Bootstrap starting.
I20260924 00:09:32.003427 1438 log.cc:824] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:32.003793 1439 log.cc:824] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:32.004034 1440 tablet_bootstrap.cc:654] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:32.004333 1438 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: No bootstrap required, opened a new log
I20260924 00:09:32.004401 1438 ts_tablet_manager.cc:1397] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:09:32.004649 1439 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: No bootstrap required, opened a new log
I20260924 00:09:32.004643 1440 log.cc:824] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:32.004710 1439 ts_tablet_manager.cc:1397] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 00:09:32.005259 1440 tablet_bootstrap.cc:492] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: No bootstrap required, opened a new log
I20260924 00:09:32.005342 1440 ts_tablet_manager.cc:1397] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:32.006351 1438 raft_consensus.cc:348] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.006541 1438 raft_consensus.cc:374] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:32.006577 1438 raft_consensus.cc:729] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae14adb618e143f0a034576cf0ab2ee7, State: Initialized, Role: FOLLOWER
I20260924 00:09:32.006523 1439 raft_consensus.cc:348] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.006680 1439 raft_consensus.cc:374] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:32.006717 1439 raft_consensus.cc:729] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a302294174485e831f0c4debdd3de2, State: Initialized, Role: FOLLOWER
I20260924 00:09:32.006685 1438 consensus_queue.cc:260] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.006986 1440 raft_consensus.cc:348] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.007139 1440 raft_consensus.cc:374] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:32.007189 1440 raft_consensus.cc:729] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c58901b450494e92d4c2d5614908b3, State: Initialized, Role: FOLLOWER
I20260924 00:09:32.006821 1439 consensus_queue.cc:260] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.007364 1440 consensus_queue.cc:260] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.007463 1438 ts_tablet_manager.cc:1428] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:32.007479 1439 ts_tablet_manager.cc:1428] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:32.007589 1153 heartbeater.cc:503] Master 127.0.98.62:33825 was elected leader, sending a full tablet report...
I20260924 00:09:32.007618 1421 heartbeater.cc:503] Master 127.0.98.62:33825 was elected leader, sending a full tablet report...
I20260924 00:09:32.007647 1440 ts_tablet_manager.cc:1428] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:09:32.007767 1287 heartbeater.cc:503] Master 127.0.98.62:33825 was elected leader, sending a full tablet report...
W20260924 00:09:32.046700 1288 tablet.cc:2369] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:32.139513 1154 tablet.cc:2369] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:32.208151 1422 tablet.cc:2369] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:32.319828 1444 raft_consensus.cc:482] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:32.319973 1444 raft_consensus.cc:504] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.320590 1444 leader_election.cc:288] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a302294174485e831f0c4debdd3de2 (127.0.98.3:43501), e1c58901b450494e92d4c2d5614908b3 (127.0.98.2:43383)
I20260924 00:09:32.325328 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.320943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46771 (local address 127.0.98.2:43383)
0924 00:09:32.320997 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:32.321000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:32.321098 (+ 98us) server_negotiation.cc:407] Connection header received
0924 00:09:32.321254 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:32.321255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:32.321268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:32.321332 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:32.321533 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.322341 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324129 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.324786 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324881 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.324893 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:32.325180 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 00:09:32.325250 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:09:32.325408 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.320957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34583 (local address 127.0.98.3:43501)
0924 00:09:32.321109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:32.321111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:32.321121 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:32.321158 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:32.321159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:32.321169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:32.321268 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:32.321509 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.322341 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324129 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.324779 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324881 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.324893 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:32.325278 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 00:09:32.325331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:32.325305 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.320856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43501 (local address 127.0.98.1:34583)
0924 00:09:32.321003 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:32.321017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:32.321077 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:32.321284 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:32.321285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:32.321325 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:32.321419 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:32.321421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.322465 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.322466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:32.323967 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:32.323971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.324911 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:32.325013 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.325015 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:32.325096 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:09:32.325156 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":84,"thread_start_us":80,"threads_started":1}
I20260924 00:09:32.325307 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.320856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43383 (local address 127.0.98.1:46771)
0924 00:09:32.321004 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:32.321017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:32.321178 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:32.321344 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:32.321344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:32.321345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:32.321424 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:32.321425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.322465 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.322466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:32.323967 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:32.323971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.324910 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.324911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:32.325008 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.325013 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:32.325140 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:09:32.325171 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":142,"thread_start_us":54,"threads_started":1}
I20260924 00:09:32.325802 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e72cde51fea4417bfce147f20fea722" candidate_uuid: "ae14adb618e143f0a034576cf0ab2ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a302294174485e831f0c4debdd3de2" is_pre_election: true
I20260924 00:09:32.325788 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e72cde51fea4417bfce147f20fea722" candidate_uuid: "ae14adb618e143f0a034576cf0ab2ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c58901b450494e92d4c2d5614908b3" is_pre_election: true
I20260924 00:09:32.325961 1241 raft_consensus.cc:2454] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae14adb618e143f0a034576cf0ab2ee7 in term 0.
I20260924 00:09:32.325961 1375 raft_consensus.cc:2454] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae14adb618e143f0a034576cf0ab2ee7 in term 0.
I20260924 00:09:32.326225 1042 leader_election.cc:302] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae14adb618e143f0a034576cf0ab2ee7, e1c58901b450494e92d4c2d5614908b3; no voters:
I20260924 00:09:32.326422 1444 raft_consensus.cc:2781] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:32.326498 1444 raft_consensus.cc:482] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:32.326517 1444 raft_consensus.cc:3037] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:32.327402 1444 raft_consensus.cc:504] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.327569 1444 leader_election.cc:288] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [CANDIDATE]: Term 1 election: Requested vote from peers e8a302294174485e831f0c4debdd3de2 (127.0.98.3:43501), e1c58901b450494e92d4c2d5614908b3 (127.0.98.2:43383)
I20260924 00:09:32.327797 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e72cde51fea4417bfce147f20fea722" candidate_uuid: "ae14adb618e143f0a034576cf0ab2ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a302294174485e831f0c4debdd3de2"
I20260924 00:09:32.327793 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e72cde51fea4417bfce147f20fea722" candidate_uuid: "ae14adb618e143f0a034576cf0ab2ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c58901b450494e92d4c2d5614908b3"
I20260924 00:09:32.327898 1375 raft_consensus.cc:3037] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:32.327894 1241 raft_consensus.cc:3037] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:32.328722 1241 raft_consensus.cc:2454] T 1e72cde51fea4417bfce147f20fea722 P e1c58901b450494e92d4c2d5614908b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae14adb618e143f0a034576cf0ab2ee7 in term 1.
I20260924 00:09:32.328747 1375 raft_consensus.cc:2454] T 1e72cde51fea4417bfce147f20fea722 P e8a302294174485e831f0c4debdd3de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae14adb618e143f0a034576cf0ab2ee7 in term 1.
I20260924 00:09:32.328943 1040 leader_election.cc:302] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae14adb618e143f0a034576cf0ab2ee7, e8a302294174485e831f0c4debdd3de2; no voters:
I20260924 00:09:32.329085 1444 raft_consensus.cc:2781] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:32.329246 1444 raft_consensus.cc:686] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [term 1 LEADER]: Becoming Leader. State: Replica: ae14adb618e143f0a034576cf0ab2ee7, State: Running, Role: LEADER
I20260924 00:09:32.329365 1444 consensus_queue.cc:237] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } }
I20260924 00:09:32.330329 960 catalog_manager.cc:5495] T 1e72cde51fea4417bfce147f20fea722 P ae14adb618e143f0a034576cf0ab2ee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae14adb618e143f0a034576cf0ab2ee7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ae14adb618e143f0a034576cf0ab2ee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a302294174485e831f0c4debdd3de2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1c58901b450494e92d4c2d5614908b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae14adb618e143f0a034576cf0ab2ee7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39011 } health_report { overall_health: HEALTHY } } }
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:32.358522 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.350544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.98.1:39011)
0924 00:09:32.350626 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:32.350628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:32.350654 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:09:32.350751 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:32.350752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:32.350765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:32.350815 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:32.351465 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.351943 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.352706 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.352890 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.352947 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.355448 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:32.355459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:32.355459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:32.355466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:32.357291 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.357673 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.357678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.357678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.357757 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.357942 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.357942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.357943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.358105 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:32.358217 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:32.358283 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 00:09:32.358363 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:32.383206 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.375119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.1:39011)
0924 00:09:32.375338 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:32.375340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:32.375351 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:32.375406 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:32.375407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:32.375417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:32.375470 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:32.376106 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.376614 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.377491 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.377693 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.377757 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.380590 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:32.380592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:32.380593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:32.380601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:32.381918 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.382321 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.382323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.382323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.382385 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.382590 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.382591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.382591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.382705 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:32.382813 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:32.383009 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 00:09:32.383097 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:32.406342 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:32.399106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.98.1:39011)
0924 00:09:32.399202 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:32.399204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:32.399214 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:32.399305 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:32.399306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:32.399316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:32.399376 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:32.399939 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.400438 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.401209 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:32.401400 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:32.401457 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:32.403784 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:32.403785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:32.403785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:32.403792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:32.405236 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.405642 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.405642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.405643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.405694 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:32.405907 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:32.405907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:32.405908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:32.406001 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:32.406075 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:32.406194 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 00:09:32.406252 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 00:09:32.406868 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1e72cde51fea4417bfce147f20fea722: Uninitialized: clean time has not yet been initialized
I20260924 00:09:32.407799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1024
I20260924 00:09:32.412890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1158
I20260924 00:09:32.418051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1292
I20260924 00:09:32.422573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 930
2026-09-24T00:09:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2273 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[1463](info): set up 2 sockets
Sep 24 00:09:32 dist-test-slave-3q8k krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260924 00:09:33.498865 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:33 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:33.621743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:33.701009 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:33.701252 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:33.701289 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:33.702427 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:33.702495 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:09:33.702521 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:33.702554 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:33.702580 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:33.702618 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:33.704198 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35483
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:33.704475 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:33.704703 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:33.709470 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:33.709506 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:33.709556 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:33.709619 1476 server_base.cc:1034] running on GCE node
I20260924 00:09:33.709872 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:33.710189 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:33.711319 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790208573711311 us; error 35 us; skew 500 ppm
Sep 24 00:09:33 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:33.719887 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:33.720358 1476 webserver.cc:460] Webserver started at http://127.0.98.62:33033/ using document root <none> and password file <none>
I20260924 00:09:33.720610 1476 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:33.720679 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:33.720835 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:33.721922 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "3db65268371f48849e614ca134dd0a4e"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "8c4500d03b34e2a57e97012e6eb594a3"
server_key_iv: "7921e4fd935b76700fccb3922a59921e"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.722277 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "3db65268371f48849e614ca134dd0a4e"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "8c4500d03b34e2a57e97012e6eb594a3"
server_key_iv: "7921e4fd935b76700fccb3922a59921e"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.723683 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:33.724450 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:33.724617 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:33.724721 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "3db65268371f48849e614ca134dd0a4e"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "8c4500d03b34e2a57e97012e6eb594a3"
server_key_iv: "7921e4fd935b76700fccb3922a59921e"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.724812 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:33.752975 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:33.754940 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:33.755127 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:33.760205 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35483
I20260924 00:09:33.760253 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35483 every 8 connection(s)
I20260924 00:09:33.760622 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:33.761312 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:33.763665 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Bootstrap starting.
I20260924 00:09:33.764272 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:33.764576 1554 log.cc:824] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:33.765262 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: No bootstrap required, opened a new log
I20260924 00:09:33.766948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1476
I20260924 00:09:33.767081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:33.767270 392 external_mini_cluster.cc:1428] Setting key a66f2afa111ec88f54bd2b04449fbe89
I20260924 00:09:33.767663 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } }
I20260924 00:09:33.767875 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:33.767941 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db65268371f48849e614ca134dd0a4e, State: Initialized, Role: FOLLOWER
I20260924 00:09:33.768079 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } }
I20260924 00:09:33.768146 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:33.768168 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:33.768240 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:33.768778 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } }
I20260924 00:09:33.768865 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db65268371f48849e614ca134dd0a4e; no voters:
I20260924 00:09:33.769047 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:33.769090 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:33.769299 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [term 1 LEADER]: Becoming Leader. State: Replica: 3db65268371f48849e614ca134dd0a4e, State: Running, Role: LEADER
I20260924 00:09:33.769320 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:33.769387 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } }
I20260924 00:09:33.769794 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db65268371f48849e614ca134dd0a4e. Latest consensus state: current_term: 1 leader_uuid: "3db65268371f48849e614ca134dd0a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } } }
I20260924 00:09:33.769826 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3db65268371f48849e614ca134dd0a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db65268371f48849e614ca134dd0a4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35483 } } }
I20260924 00:09:33.769935 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:33.770123 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:33.770277 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:33.770937 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:33.772709 1568 catalog_manager.cc:1347] Generated new cluster ID: b8e36081b6264a60a4e0fcc36d1173c4
I20260924 00:09:33.772760 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:33 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:33.781512 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:33.767782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.98.62:35483)
0924 00:09:33.767999 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:33.768009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:33.768040 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:09:33.768193 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:33.768201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:33.768288 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:33.768443 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:33.769175 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.770559 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.771607 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.772201 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.772308 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:33.777569 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:33.777573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:33.777574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:33.777587 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:33.779648 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.780041 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.780044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.780045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.780143 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.780311 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.780312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.780312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.780604 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:33.780751 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:33.781141 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 00:09:33.781272 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260924 00:09:33.798087 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:33.798620 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:33.808318 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Generated new TSK 0
I20260924 00:09:33.808531 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:33.837977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35483
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:33.925861 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:33.926074 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:33.926101 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:33.927347 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:33.927392 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:33.927413 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:33.927431 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:33.929091 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:33.929337 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:33.929600 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:33.934470 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:33.934469 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:33.934527 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:33.934649 1580 server_base.cc:1034] running on GCE node
I20260924 00:09:33.934960 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:33.935258 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:33.936419 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790208573936409 us; error 31 us; skew 500 ppm
Sep 24 00:09:33 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:33.938894 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:33.939447 1580 webserver.cc:460] Webserver started at http://127.0.98.1:33409/ using document root <none> and password file <none>
I20260924 00:09:33.939667 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:33.939735 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:33.939882 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:33.940927 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "41c900097e464c4ca2014e94ffe0c09f"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "65c3f38fb969f985d926ed18bad5fc61"
server_key_iv: "b9c6670cfb019828433ea9a2c770b77b"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.941290 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "41c900097e464c4ca2014e94ffe0c09f"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "65c3f38fb969f985d926ed18bad5fc61"
server_key_iv: "b9c6670cfb019828433ea9a2c770b77b"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.942667 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:33.943507 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:33.943670 1580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:33.943771 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "41c900097e464c4ca2014e94ffe0c09f"
format_stamp: "Formatted at 2026-09-24 00:09:33 on dist-test-slave-3q8k"
server_key: "65c3f38fb969f985d926ed18bad5fc61"
server_key_iv: "b9c6670cfb019828433ea9a2c770b77b"
server_key_version: "encryptionkey@0"
I20260924 00:09:33.943842 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:33.952375 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:33.954699 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:33.954856 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:33.955443 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:33.955505 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:33.955538 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:33.955569 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:33.962050 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36399
I20260924 00:09:33.962158 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36399 every 8 connection(s)
Sep 24 00:09:33 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:33.962453 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:33.963106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1580
I20260924 00:09:33.963236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:33.963429 392 external_mini_cluster.cc:1428] Setting key 4fe9d9a59343d3aff30cc73290ffd64b
I20260924 00:09:33.966046 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:33.955905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58389 (local address 127.0.98.62:35483)
0924 00:09:33.955961 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:33.955964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:33.956175 (+ 211us) server_negotiation.cc:407] Connection header received
0924 00:09:33.956645 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:33.956646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:33.956663 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:33.956747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:33.957593 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.958420 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.960033 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.960376 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.960460 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:33.962916 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:33.962918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:33.962919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:33.962929 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:33.964581 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.964957 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.964959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.964959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.965020 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.965283 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.965284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.965284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.965429 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:33.965495 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:33.965875 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 00:09:33.965957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:09:33.966235 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:33.955877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.1:58389)
0924 00:09:33.956035 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:33.956073 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:33.956546 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:33.956797 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:33.956800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:33.957169 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:33.957476 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:33.957483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.958520 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.958521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:33.959922 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:33.959925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.960458 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.960459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:33.960552 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:33.961089 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:33.961100 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:33.962804 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:33.964670 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:33.964676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:33.964677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:33.964885 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:33.965123 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:33.965123 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:33.965123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:33.965218 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:33.965530 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:33.965538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:33.965720 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 00:09:33.965840 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 00:09:33.965997 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 00:09:33.966778 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35483
I20260924 00:09:33.966883 1711 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:33.967068 1711 heartbeater.cc:511] Master 127.0.98.62:35483 requested a full tablet report, sending...
I20260924 00:09:33.967602 1508 ts_manager.cc:194] Registered new tserver with Master: 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399)
I20260924 00:09:33.968508 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58389
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:09:33.974804 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:33.967912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.1:57547)
0924 00:09:33.968057 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:33.968068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:33.968129 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:33.968263 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:33.968264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:33.968439 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:33.968497 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:33.968498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.969486 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.969487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:33.970314 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:33.970317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.970798 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.970799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:33.970851 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:33.971436 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:33.971441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:33.971754 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:33.973437 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:33.973438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:33.973438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:33.973726 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:33.973939 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:33.973940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:33.973941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:33.974024 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:33.974465 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:33.974465 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:33.974508 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 00:09:33.974598 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:09:33.974703 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":48,"thread_start_us":80,"threads_started":1}
I20260924 00:09:33.974835 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:33.967842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57547 (local address 127.0.98.62:35483)
0924 00:09:33.967880 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:33.967882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:33.968139 (+ 257us) server_negotiation.cc:407] Connection header received
0924 00:09:33.968172 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:33.968173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:33.968192 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:33.968259 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:33.968582 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.969389 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.970409 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:33.970710 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:33.970792 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:33.971841 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:33.971843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:33.971843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:33.971850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:33.973310 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.973805 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.973806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.973807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.973868 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:33.974103 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:33.974129 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:33.974129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:33.974361 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:33.974435 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:33.974657 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 00:09:33.974749 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:33.990296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35483
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:34.074136 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:34.074369 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:34.074410 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:34.075620 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:34.075685 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:34.075717 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:34.075755 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:34.077251 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:34.077517 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:34.077757 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:34.082886 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:34.082899 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:34.082899 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:34.083240 1720 server_base.cc:1034] running on GCE node
I20260924 00:09:34.083482 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:34.083736 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:34.084926 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790208574084924 us; error 39 us; skew 500 ppm
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:34.087720 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:34.088173 1720 webserver.cc:460] Webserver started at http://127.0.98.2:43827/ using document root <none> and password file <none>
I20260924 00:09:34.088403 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:34.088471 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:34.088645 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:34.089781 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b730ad1cdfe49ed9700fe867db9f67d"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "89dee9cf9c6f080137d8a9a45686025b"
server_key_iv: "5f0c0f49fb10a15b5368c9b5499736bf"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.090143 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b730ad1cdfe49ed9700fe867db9f67d"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "89dee9cf9c6f080137d8a9a45686025b"
server_key_iv: "5f0c0f49fb10a15b5368c9b5499736bf"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.091566 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:09:34.092330 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.092504 1720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:34.092640 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "2b730ad1cdfe49ed9700fe867db9f67d"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "89dee9cf9c6f080137d8a9a45686025b"
server_key_iv: "5f0c0f49fb10a15b5368c9b5499736bf"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.092717 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:34.105966 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:34.108132 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:34.108300 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:34.108814 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:34.108877 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.108922 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:34.108953 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.116546 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34431
I20260924 00:09:34.116638 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34431 every 8 connection(s)
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:34.116966 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:34.120045 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.109436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59233 (local address 127.0.98.62:35483)
0924 00:09:34.109510 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.109513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.109741 (+ 228us) server_negotiation.cc:407] Connection header received
0924 00:09:34.110535 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.110536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.110550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.110645 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.111577 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.112380 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.114058 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.114362 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.114469 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.117304 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.117305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.117306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.117314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.118678 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.119076 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.119077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.119078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.119135 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.119412 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.119413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.119413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.119526 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.119606 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.119874 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.119956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:09:34.120221 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.109404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.2:59233)
0924 00:09:34.109595 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.109645 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.110385 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.110732 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.110738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.111154 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:34.111469 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.111475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.112478 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.112479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.113956 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.113960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.114473 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.114474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.114617 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.115314 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:34.115326 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:34.117212 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:34.118763 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.118769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.118771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.118986 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.119209 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.119209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.119210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.119342 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.119602 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:34.119608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:34.119742 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.119839 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.119989 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260924 00:09:34.120711 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35483
I20260924 00:09:34.120800 1851 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:34.120987 1851 heartbeater.cc:511] Master 127.0.98.62:35483 requested a full tablet report, sending...
I20260924 00:09:34.121501 1507 ts_manager.cc:194] Registered new tserver with Master: 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2:34431)
I20260924 00:09:34.121979 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59233
I20260924 00:09:34.125383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1720
I20260924 00:09:34.125506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:34.125684 392 external_mini_cluster.cc:1428] Setting key a3f4c3e5b645222b1df2838e7cac2871
I20260924 00:09:34.127475 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.121746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58139 (local address 127.0.98.62:35483)
0924 00:09:34.121780 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.121782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.121908 (+ 126us) server_negotiation.cc:407] Connection header received
0924 00:09:34.121979 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.121980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.121990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.122038 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.122377 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.122847 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.123633 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.123832 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.123914 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.124717 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.124718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.124718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.124725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.126252 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.126609 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.126610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.126610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.126673 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.126884 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.126885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.126885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.127052 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.127127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.127356 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.127415 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 00:09:34.127699 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.121710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.2:58139)
0924 00:09:34.121831 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.121841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.121923 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.122040 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.122040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.122253 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:34.122301 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.122302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.122939 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.122940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.123550 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.123552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.123916 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.123917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.123978 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.124368 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:34.124372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:34.124632 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:34.126338 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.126339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.126339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.126539 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.126734 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.126735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.126736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.126806 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.127141 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:34.127142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:34.127192 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.127335 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.127447 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:34.150513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35483
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:34.231977 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:34.232184 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:34.232208 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:34.233309 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:34.233345 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:34.233366 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:34.233382 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:34.234807 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:34.235054 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:34.235289 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:34.239948 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:34.239938 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:34.239938 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:34.240250 1860 server_base.cc:1034] running on GCE node
I20260924 00:09:34.240460 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:34.240690 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:34.241846 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790208574241838 us; error 33 us; skew 500 ppm
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:34.244472 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:34.244915 1860 webserver.cc:460] Webserver started at http://127.0.98.3:42701/ using document root <none> and password file <none>
I20260924 00:09:34.245158 1860 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:34.245206 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:34.245343 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:34.246449 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "23dd5c1cab824af996193d8d8762e7f5"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "f04829252c05e4eb632475330ea572de"
server_key_iv: "b94c825308f0aa4f5a13d937ef0f66c1"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.246891 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23dd5c1cab824af996193d8d8762e7f5"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "f04829252c05e4eb632475330ea572de"
server_key_iv: "b94c825308f0aa4f5a13d937ef0f66c1"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.248425 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:34.249226 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.249440 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:34.249545 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "23dd5c1cab824af996193d8d8762e7f5"
format_stamp: "Formatted at 2026-09-24 00:09:34 on dist-test-slave-3q8k"
server_key: "f04829252c05e4eb632475330ea572de"
server_key_iv: "b94c825308f0aa4f5a13d937ef0f66c1"
server_key_version: "encryptionkey@0"
I20260924 00:09:34.249614 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:34.263718 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:34.265743 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:34.265908 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:34.266525 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:34.266582 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.266616 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:34.266634 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:34.274735 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39005
I20260924 00:09:34.275089 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:34.275135 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39005 every 8 connection(s)
I20260924 00:09:34.275520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 1860
I20260924 00:09:34.275616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:34.275789 392 external_mini_cluster.cc:1428] Setting key da62030f062fcec1490e5f19248f58f4
I20260924 00:09:34.277210 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.267002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38265 (local address 127.0.98.62:35483)
0924 00:09:34.267054 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.267057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.267323 (+ 266us) server_negotiation.cc:407] Connection header received
0924 00:09:34.268069 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.268070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.268082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.268177 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.269163 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.269929 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.271396 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.271651 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.271745 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.274033 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.274035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.274036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.274044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.275771 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.276241 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.276243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.276243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.276302 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.276610 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.276612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.276612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.276767 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.276822 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.277056 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.277119 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:34.277346 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.266962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.3:38265)
0924 00:09:34.267125 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.267235 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.267939 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.268239 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.268244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.268641 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:34.268995 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.269003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.270090 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.270091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.271323 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.271326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.271765 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.271765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.271892 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.272476 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:34.272486 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:34.273937 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:34.275863 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.275869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.275871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.276161 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.276376 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.276377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.276377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.276499 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.276830 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:34.276834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:34.276943 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.277027 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.277162 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260924 00:09:34.277742 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35483
I20260924 00:09:34.277823 1991 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:34.278009 1991 heartbeater.cc:511] Master 127.0.98.62:35483 requested a full tablet report, sending...
I20260924 00:09:34.278386 1507 ts_manager.cc:194] Registered new tserver with Master: 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.278834 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38265
I20260924 00:09:34.284060 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.278812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39889 (local address 127.0.98.62:35483)
0924 00:09:34.278845 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.278847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.278922 (+ 75us) server_negotiation.cc:407] Connection header received
0924 00:09:34.279017 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.279018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.279027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.279100 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.279891 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.280345 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.281058 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.281230 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.281280 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.282108 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.282110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.282110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.282115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.283203 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.283536 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.283537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.283537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.283584 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.283713 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.283713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.283713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.283799 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.283847 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.283952 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.284004 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 00:09:34.284062 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.278742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.98.3:39889)
0924 00:09:34.278862 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.278872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.278948 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.279109 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.279110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.279770 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:34.279814 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.279816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.280459 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.280460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.280986 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.280987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.281316 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.281317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.281392 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.281769 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:34.281773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:34.282035 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:34.283325 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.283325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.283326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.283478 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.283627 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.283627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.283627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.283667 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.283845 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:34.283846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:34.283875 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.283923 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.283991 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260924 00:09:34.287485 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:34.318517 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.310538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.98.62:35483)
0924 00:09:34.310613 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.310616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.310632 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:09:34.310668 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.310669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.310678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.310739 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.311619 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.312265 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.313074 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.313258 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.313305 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.315959 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.315961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.315961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.315970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.317350 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.317734 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.317736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.317736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.317796 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.317997 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.317998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.317998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.318115 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.318217 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.318327 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.318407 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:09:34.320241 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:34.322039 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:34.328455 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.0.1:47760)
0924 00:09:34.324615 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.324626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.324677 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.325135 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.325136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.325138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.325173 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.325175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.326072 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.326073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.327322 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.327325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328136 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.328229 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.328243 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.328315 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.328367 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260924 00:09:34.328461 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.98.3:39005)
0924 00:09:34.324960 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.324963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.324972 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:34.325006 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.325009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.325024 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.325130 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.325258 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.325980 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.327416 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328044 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328130 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.328210 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.328340 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.328376 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":71,"threads_started":1}
I20260924 00:09:34.328650 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36399 (local address 127.0.0.1:37424)
0924 00:09:34.324544 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.324582 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.324649 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.325291 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.325291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.325293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.325331 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.325333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.326540 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.326542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.327435 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.327437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328397 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.328469 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.328473 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.328504 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.328548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":54,"threads_started":1}
I20260924 00:09:34.329015 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.0.98.1:36399)
0924 00:09:34.324911 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.324916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.324926 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:34.324968 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.324970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.324987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.325411 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.325465 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.326475 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.327516 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328262 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328753 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.328834 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.328859 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.328900 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260924 00:09:34.329172 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34431 (local address 127.0.0.1:35744)
0924 00:09:34.324812 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.324822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.324873 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.325079 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.325083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.325085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.325156 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.325158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.326463 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.326464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.327657 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.327660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328920 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.328984 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.328987 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.329034 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.329075 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260924 00:09:34.329162 1919 tablet_service.cc:1467] Processing CreateTablet for tablet 66f46243692242e5935bef0331fc3801 (DEFAULT_TABLE table=test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.329517 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.324659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.0.98.2:34431)
0924 00:09:34.324804 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.324810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.324862 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:09:34.324937 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.324939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.324953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.325051 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.325227 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.326373 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.327737 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.328415 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.328471 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.329340 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.329369 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.329411 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260924 00:09:34.329540 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f46243692242e5935bef0331fc3801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.329679 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 66f46243692242e5935bef0331fc3801 (DEFAULT_TABLE table=test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.329995 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f46243692242e5935bef0331fc3801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.333421 2014 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Bootstrap starting.
I20260924 00:09:34.333853 2015 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Bootstrap starting.
I20260924 00:09:34.333963 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 66f46243692242e5935bef0331fc3801 (DEFAULT_TABLE table=test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.334244 2014 tablet_bootstrap.cc:654] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.334336 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f46243692242e5935bef0331fc3801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.334461 2015 tablet_bootstrap.cc:654] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.334580 2014 log.cc:824] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:34.334776 2015 log.cc:824] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:34.335355 2014 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: No bootstrap required, opened a new log
I20260924 00:09:34.335433 2014 ts_tablet_manager.cc:1397] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:34.336923 2014 raft_consensus.cc:348] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.337072 2014 raft_consensus.cc:374] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.337117 2014 raft_consensus.cc:729] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dd5c1cab824af996193d8d8762e7f5, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.337240 2014 consensus_queue.cc:260] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.337540 2014 ts_tablet_manager.cc:1428] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:34.337574 1991 heartbeater.cc:503] Master 127.0.98.62:35483 was elected leader, sending a full tablet report...
I20260924 00:09:34.337863 2019 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Bootstrap starting.
I20260924 00:09:34.338177 2015 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: No bootstrap required, opened a new log
I20260924 00:09:34.338258 2015 ts_tablet_manager.cc:1397] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:09:34.338560 2019 tablet_bootstrap.cc:654] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.338874 2019 log.cc:824] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:34.339807 2019 tablet_bootstrap.cc:492] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: No bootstrap required, opened a new log
I20260924 00:09:34.339870 2019 ts_tablet_manager.cc:1397] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:34.340459 2015 raft_consensus.cc:348] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.340624 2015 raft_consensus.cc:374] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.340663 2015 raft_consensus.cc:729] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c900097e464c4ca2014e94ffe0c09f, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.340776 2015 consensus_queue.cc:260] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.341027 2015 ts_tablet_manager.cc:1428] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:09:34.341135 1711 heartbeater.cc:503] Master 127.0.98.62:35483 was elected leader, sending a full tablet report...
I20260924 00:09:34.341830 2019 raft_consensus.cc:348] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.341943 2019 raft_consensus.cc:374] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.341964 2019 raft_consensus.cc:729] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b730ad1cdfe49ed9700fe867db9f67d, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.342087 2019 consensus_queue.cc:260] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.342326 2019 ts_tablet_manager.cc:1428] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:09:34.342377 1851 heartbeater.cc:503] Master 127.0.98.62:35483 was elected leader, sending a full tablet report...
W20260924 00:09:34.367530 1852 tablet.cc:2369] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:34.463379 1712 tablet.cc:2369] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:34.526237 1992 tablet.cc:2369] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:34.547022 2022 raft_consensus.cc:482] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:34.547153 2022 raft_consensus.cc:504] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.547612 2022 leader_election.cc:288] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.550745 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.547841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.2:43893)
0924 00:09:34.547884 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.547896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.547941 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.548176 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.548177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.548179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.548274 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.548276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.548960 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.548961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.549916 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.549918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.550447 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.550448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.550533 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.550536 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.550596 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.550644 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":46}
I20260924 00:09:34.550853 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.547888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50137 (local address 127.0.98.1:36399)
0924 00:09:34.547937 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.547939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.548098 (+ 159us) server_negotiation.cc:407] Connection header received
0924 00:09:34.548150 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.548151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.548168 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.548224 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.548387 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.548868 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.549880 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.550492 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.550563 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.550573 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.550742 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.550774 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:34.550861 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.547841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36399 (local address 127.0.98.2:50137)
0924 00:09:34.548043 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.548051 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.548098 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.548237 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.548238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.548239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.548305 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.548307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.548951 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.548952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.549777 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.549779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.550606 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.550607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.550679 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.550684 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.550737 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.550774 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":50,"thread_start_us":108,"threads_started":1}
I20260924 00:09:34.551085 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.547971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43893 (local address 127.0.98.3:39005)
0924 00:09:34.548033 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.548035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.548050 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:09:34.548075 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.548076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.548085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.548171 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.548375 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.548865 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.550001 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.550364 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.550449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.550459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.550977 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.551028 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:09:34.551183 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f46243692242e5935bef0331fc3801" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c900097e464c4ca2014e94ffe0c09f" is_pre_election: true
I20260924 00:09:34.551365 1665 raft_consensus.cc:2454] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 0.
I20260924 00:09:34.551370 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f46243692242e5935bef0331fc3801" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5" is_pre_election: true
I20260924 00:09:34.551522 1945 raft_consensus.cc:2454] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 0.
I20260924 00:09:34.551604 1736 leader_election.cc:302] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b730ad1cdfe49ed9700fe867db9f67d, 41c900097e464c4ca2014e94ffe0c09f; no voters:
I20260924 00:09:34.551724 2022 raft_consensus.cc:2781] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:34.551779 2022 raft_consensus.cc:482] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:34.551811 2022 raft_consensus.cc:3037] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.552906 2022 raft_consensus.cc:504] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.553048 2022 leader_election.cc:288] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 election: Requested vote from peers 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.553212 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f46243692242e5935bef0331fc3801" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c900097e464c4ca2014e94ffe0c09f"
I20260924 00:09:34.553212 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f46243692242e5935bef0331fc3801" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5"
I20260924 00:09:34.553288 1945 raft_consensus.cc:3037] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.553288 1665 raft_consensus.cc:3037] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.554149 1945 raft_consensus.cc:2454] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 1.
I20260924 00:09:34.554149 1665 raft_consensus.cc:2454] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 1.
I20260924 00:09:34.554315 1736 leader_election.cc:302] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b730ad1cdfe49ed9700fe867db9f67d, 41c900097e464c4ca2014e94ffe0c09f; no voters:
I20260924 00:09:34.554435 2022 raft_consensus.cc:2781] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:34.554569 2022 raft_consensus.cc:686] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 LEADER]: Becoming Leader. State: Replica: 2b730ad1cdfe49ed9700fe867db9f67d, State: Running, Role: LEADER
I20260924 00:09:34.554673 2022 consensus_queue.cc:237] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } }
I20260924 00:09:34.555583 1507 catalog_manager.cc:5495] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } health_report { overall_health: HEALTHY } } }
I20260924 00:09:34.611888 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.609345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.0.98.2:34431)
0924 00:09:34.609410 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.609411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.609418 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:09:34.609449 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.609450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.609467 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.609542 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:34.609704 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.610177 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.610995 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.611186 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.611330 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.611547 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.611653 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.611744 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.611784 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.611804 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:09:34.613758 1665 raft_consensus.cc:1261] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Refusing update from remote peer 2b730ad1cdfe49ed9700fe867db9f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.613765 1945 raft_consensus.cc:1261] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Refusing update from remote peer 2b730ad1cdfe49ed9700fe867db9f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.614154 2024 consensus_queue.cc:1035] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.614315 2022 consensus_queue.cc:1035] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.616433 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332694321614974976 to 7332694321376669696. Current Snapshot: MvccSnapshot[applied={T|T < 7332694321614974976}]
I20260924 00:09:34.616699 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332694321614974976 to 7332694321376669696. Current Snapshot: MvccSnapshot[applied={T|T < 7332694321614974976}]
I20260924 00:09:34.617008 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332694321614974976 to 7332694321376669696. Current Snapshot: MvccSnapshot[applied={T|T < 7332694321614974976}]
I20260924 00:09:34.620240 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:34.620352 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:34.621065 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Sending DeleteTablet for 3 replicas of tablet 66f46243692242e5935bef0331fc3801
I20260924 00:09:34.621424 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f46243692242e5935bef0331fc3801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35744
I20260924 00:09:34.621400 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f46243692242e5935bef0331fc3801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37424
I20260924 00:09:34.621738 2038 tablet_replica.cc:330] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: stopping tablet replica
I20260924 00:09:34.621781 1919 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f46243692242e5935bef0331fc3801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47760
I20260924 00:09:34.621837 2038 raft_consensus.cc:2229] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:34.621980 2038 raft_consensus.cc:2258] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.621971 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:34.622113 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:34.622258 2037 tablet_replica.cc:330] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: stopping tablet replica
I20260924 00:09:34.622745 2038 ts_tablet_manager.cc:1905] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.623452 1919 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb1fc812093445cbb55accfacba597a (DEFAULT_TABLE table=test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.623451 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb1fc812093445cbb55accfacba597a (DEFAULT_TABLE table=test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.623564 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb1fc812093445cbb55accfacba597a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.623564 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb1fc812093445cbb55accfacba597a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.624831 2014 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Bootstrap starting.
I20260924 00:09:34.624837 2019 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Bootstrap starting.
I20260924 00:09:34.625276 2019 tablet_bootstrap.cc:654] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.625278 2014 tablet_bootstrap.cc:654] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.626027 2039 tablet_replica.cc:330] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: stopping tablet replica
I20260924 00:09:34.626155 2039 raft_consensus.cc:2229] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:34.622363 2037 raft_consensus.cc:2229] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:34.624919 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb1fc812093445cbb55accfacba597a (DEFAULT_TABLE table=test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:34.626330 2039 raft_consensus.cc:2258] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.626360 2037 raft_consensus.cc:2258] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.626387 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb1fc812093445cbb55accfacba597a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.626820 2039 ts_tablet_manager.cc:1905] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.626930 2014 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: No bootstrap required, opened a new log
I20260924 00:09:34.626966 2014 ts_tablet_manager.cc:1397] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:34.627103 2014 raft_consensus.cc:348] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.627195 2014 raft_consensus.cc:374] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.627234 2014 raft_consensus.cc:729] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dd5c1cab824af996193d8d8762e7f5, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.627278 2014 consensus_queue.cc:260] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.627367 2014 ts_tablet_manager.cc:1428] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.628098 2015 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Bootstrap starting.
I20260924 00:09:34.628314 2039 ts_tablet_manager.cc:1918] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:34.628328 2019 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: No bootstrap required, opened a new log
I20260924 00:09:34.628366 2019 ts_tablet_manager.cc:1397] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:09:34.628379 2039 log.cc:1196] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/66f46243692242e5935bef0331fc3801
I20260924 00:09:34.628468 2015 tablet_bootstrap.cc:654] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.628489 2019 raft_consensus.cc:348] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.628563 2019 raft_consensus.cc:374] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.628587 2019 raft_consensus.cc:729] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b730ad1cdfe49ed9700fe867db9f67d, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.628650 2039 ts_tablet_manager.cc:1939] T 66f46243692242e5935bef0331fc3801 P 23dd5c1cab824af996193d8d8762e7f5: Deleting consensus metadata
I20260924 00:09:34.628646 2019 consensus_queue.cc:260] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.628757 2019 ts_tablet_manager.cc:1428] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:34.629051 2038 ts_tablet_manager.cc:1918] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:34.629096 1493 catalog_manager.cc:4841] TS 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005): tablet 66f46243692242e5935bef0331fc3801 (table test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]) successfully deleted
I20260924 00:09:34.629128 2038 log.cc:1196] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/66f46243692242e5935bef0331fc3801
I20260924 00:09:34.629402 2037 ts_tablet_manager.cc:1905] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.629429 2038 ts_tablet_manager.cc:1939] T 66f46243692242e5935bef0331fc3801 P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting consensus metadata
I20260924 00:09:34.629851 1495 catalog_manager.cc:4841] TS 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2:34431): tablet 66f46243692242e5935bef0331fc3801 (table test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]) successfully deleted
I20260924 00:09:34.630833 2037 ts_tablet_manager.cc:1918] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:34.630918 2037 log.cc:1196] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/66f46243692242e5935bef0331fc3801
I20260924 00:09:34.631165 2037 ts_tablet_manager.cc:1939] T 66f46243692242e5935bef0331fc3801 P 41c900097e464c4ca2014e94ffe0c09f: Deleting consensus metadata
I20260924 00:09:34.631558 1492 catalog_manager.cc:4841] TS 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399): tablet 66f46243692242e5935bef0331fc3801 (table test-table [id=1791c4a1bbd34e42b9c26705aa86aba9]) successfully deleted
I20260924 00:09:34.635650 2015 tablet_bootstrap.cc:492] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: No bootstrap required, opened a new log
I20260924 00:09:34.635713 2015 ts_tablet_manager.cc:1397] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260924 00:09:34.635859 2015 raft_consensus.cc:348] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.635939 2015 raft_consensus.cc:374] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.635962 2015 raft_consensus.cc:729] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c900097e464c4ca2014e94ffe0c09f, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.636004 2015 consensus_queue.cc:260] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.636327 2015 ts_tablet_manager.cc:1428] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:34.778019 2021 raft_consensus.cc:482] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:34.778134 2021 raft_consensus.cc:504] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.778496 2021 leader_election.cc:288] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2:34431), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.781494 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.778772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35219 (local address 127.0.98.3:39005)
0924 00:09:34.778823 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.778826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.778905 (+ 79us) server_negotiation.cc:407] Connection header received
0924 00:09:34.778976 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.778977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.778986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.779060 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.779247 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.779721 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.780778 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.781133 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.781215 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.781224 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.781384 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.781422 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:09:34.781529 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.778858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43427 (local address 127.0.98.2:34431)
0924 00:09:34.778915 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.778918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.778944 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:09:34.778976 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.778977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.778985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.779063 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:34.779245 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.779722 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.780776 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.781151 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.781234 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.781245 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.781422 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.781459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:09:34.781538 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.778673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39005 (local address 127.0.98.1:35219)
0924 00:09:34.778834 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.778845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.778911 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.779077 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.779077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.779080 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.779162 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.779164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.779821 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.779822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.780669 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.780671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.781215 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.781216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.781298 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.781301 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.781366 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.781407 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":37,"thread_start_us":57,"threads_started":1}
I20260924 00:09:34.781551 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.778673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34431 (local address 127.0.98.1:43427)
0924 00:09:34.778819 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.778833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.778900 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.779067 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.779068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.779070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:34.779141 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.779144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.779815 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.779816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.780679 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.780681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.781262 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.781263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.781335 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.781338 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.781414 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.781454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":53,"thread_start_us":84,"threads_started":1}
I20260924 00:09:34.781793 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb1fc812093445cbb55accfacba597a" candidate_uuid: "41c900097e464c4ca2014e94ffe0c09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5" is_pre_election: true
I20260924 00:09:34.781883 1945 raft_consensus.cc:2454] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c900097e464c4ca2014e94ffe0c09f in term 0.
I20260924 00:09:34.781868 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb1fc812093445cbb55accfacba597a" candidate_uuid: "41c900097e464c4ca2014e94ffe0c09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" is_pre_election: true
I20260924 00:09:34.782001 1805 raft_consensus.cc:2454] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c900097e464c4ca2014e94ffe0c09f in term 0.
I20260924 00:09:34.782061 1597 leader_election.cc:302] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dd5c1cab824af996193d8d8762e7f5, 41c900097e464c4ca2014e94ffe0c09f; no voters:
I20260924 00:09:34.782218 2021 raft_consensus.cc:2781] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:34.782269 2021 raft_consensus.cc:482] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:34.782287 2021 raft_consensus.cc:3037] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.783650 2021 raft_consensus.cc:504] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.783843 2021 leader_election.cc:288] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [CANDIDATE]: Term 1 election: Requested vote from peers 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2:34431), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.784004 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb1fc812093445cbb55accfacba597a" candidate_uuid: "41c900097e464c4ca2014e94ffe0c09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b730ad1cdfe49ed9700fe867db9f67d"
I20260924 00:09:34.784029 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb1fc812093445cbb55accfacba597a" candidate_uuid: "41c900097e464c4ca2014e94ffe0c09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5"
I20260924 00:09:34.784082 1805 raft_consensus.cc:3037] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.784111 1945 raft_consensus.cc:3037] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.784736 1805 raft_consensus.cc:2454] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c900097e464c4ca2014e94ffe0c09f in term 1.
I20260924 00:09:34.784825 1945 raft_consensus.cc:2454] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c900097e464c4ca2014e94ffe0c09f in term 1.
I20260924 00:09:34.784915 1599 leader_election.cc:302] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b730ad1cdfe49ed9700fe867db9f67d, 41c900097e464c4ca2014e94ffe0c09f; no voters:
I20260924 00:09:34.785001 2021 raft_consensus.cc:2781] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:34.785131 2021 raft_consensus.cc:686] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 1 LEADER]: Becoming Leader. State: Replica: 41c900097e464c4ca2014e94ffe0c09f, State: Running, Role: LEADER
I20260924 00:09:34.785188 2021 consensus_queue.cc:237] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.785869 1508 catalog_manager.cc:5495] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "41c900097e464c4ca2014e94ffe0c09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:34.830806 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.823856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35483 (local address 127.0.0.1:50206)
0924 00:09:34.824033 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.824045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.824134 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.824287 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.824288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.824575 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:34.824620 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.824622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.825274 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.825275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.825796 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.825798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.826112 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.826113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.826179 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.826644 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:34.826648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:34.828114 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:34.829804 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.829810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.829812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.830017 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.830250 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:34.830251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:34.830251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:34.830308 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:34.830513 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:34.830519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:34.830559 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.830610 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.830710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260924 00:09:34.830860 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.823957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.98.62:35483)
0924 00:09:34.824121 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.824123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.824133 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:34.824196 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.824197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.824205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.824273 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.824683 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.825183 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.825868 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.826050 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.826098 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.828204 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.828206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.828206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.828212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.829699 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.830126 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.830127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.830127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.830189 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.830363 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.830363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.830364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.830462 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.830511 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.830628 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.830759 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260924 00:09:34.832901 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50206:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:09:34.833122 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:34.834563 1919 tablet_service.cc:1467] Processing CreateTablet for tablet a781b423afed442a99e785cf45b1903b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8837a921b4455aaae039f0a9e48b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:34.834561 1785 tablet_service.cc:1467] Processing CreateTablet for tablet a781b423afed442a99e785cf45b1903b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8837a921b4455aaae039f0a9e48b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:34.834595 1645 tablet_service.cc:1467] Processing CreateTablet for tablet a781b423afed442a99e785cf45b1903b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8837a921b4455aaae039f0a9e48b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:34.834721 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a781b423afed442a99e785cf45b1903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.834720 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a781b423afed442a99e785cf45b1903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.834725 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a781b423afed442a99e785cf45b1903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:34.836107 2014 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: Bootstrap starting.
I20260924 00:09:34.836421 2015 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: Bootstrap starting.
I20260924 00:09:34.836433 2019 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: Bootstrap starting.
I20260924 00:09:34.836512 2014 tablet_bootstrap.cc:654] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.836769 2019 tablet_bootstrap.cc:654] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.836846 2015 tablet_bootstrap.cc:654] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:34.837381 2014 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: No bootstrap required, opened a new log
I20260924 00:09:34.837373 2019 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: No bootstrap required, opened a new log
I20260924 00:09:34.837438 2014 ts_tablet_manager.cc:1397] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:34.837440 2019 ts_tablet_manager.cc:1397] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:34.837554 2015 tablet_bootstrap.cc:492] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: No bootstrap required, opened a new log
I20260924 00:09:34.837590 2015 ts_tablet_manager.cc:1397] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:34.837570 2014 raft_consensus.cc:348] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837553 2019 raft_consensus.cc:348] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837620 2014 raft_consensus.cc:374] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.837623 2019 raft_consensus.cc:374] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.837634 2014 raft_consensus.cc:729] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dd5c1cab824af996193d8d8762e7f5, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.837651 2019 raft_consensus.cc:729] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b730ad1cdfe49ed9700fe867db9f67d, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.837697 2019 consensus_queue.cc:260] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837700 2014 consensus_queue.cc:260] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837718 2015 raft_consensus.cc:348] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837785 2019 ts_tablet_manager.cc:1428] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:34.837790 2015 raft_consensus.cc:374] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:34.837810 2014 ts_tablet_manager.cc:1428] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:34.837817 2015 raft_consensus.cc:729] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c900097e464c4ca2014e94ffe0c09f, State: Initialized, Role: FOLLOWER
I20260924 00:09:34.837859 2015 consensus_queue.cc:260] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.837936 2015 ts_tablet_manager.cc:1428] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:34.837919 2032 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.838006 2032 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.838001 2021 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.838083 2021 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.838151 2018 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.838230 2018 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.842078 2032 raft_consensus.cc:482] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:34.842161 2032 raft_consensus.cc:504] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.842329 2032 leader_election.cc:288] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.842504 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a781b423afed442a99e785cf45b1903b" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5" is_pre_election: true
I20260924 00:09:34.842541 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a781b423afed442a99e785cf45b1903b" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c900097e464c4ca2014e94ffe0c09f" is_pre_election: true
I20260924 00:09:34.842617 1665 raft_consensus.cc:2454] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 0.
I20260924 00:09:34.842603 1945 raft_consensus.cc:2454] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 0.
I20260924 00:09:34.842754 1737 leader_election.cc:302] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dd5c1cab824af996193d8d8762e7f5, 2b730ad1cdfe49ed9700fe867db9f67d; no voters:
I20260924 00:09:34.842880 2032 raft_consensus.cc:2781] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:34.842914 2032 raft_consensus.cc:482] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:34.842944 2032 raft_consensus.cc:3037] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.843650 2032 raft_consensus.cc:504] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.843794 2032 leader_election.cc:288] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 election: Requested vote from peers 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399), 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005)
I20260924 00:09:34.843981 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a781b423afed442a99e785cf45b1903b" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23dd5c1cab824af996193d8d8762e7f5"
I20260924 00:09:34.844077 1945 raft_consensus.cc:3037] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.844148 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a781b423afed442a99e785cf45b1903b" candidate_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c900097e464c4ca2014e94ffe0c09f"
I20260924 00:09:34.844226 1665 raft_consensus.cc:3037] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:34.844784 1945 raft_consensus.cc:2454] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 1.
I20260924 00:09:34.844790 1665 raft_consensus.cc:2454] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b730ad1cdfe49ed9700fe867db9f67d in term 1.
I20260924 00:09:34.844973 1737 leader_election.cc:302] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dd5c1cab824af996193d8d8762e7f5, 2b730ad1cdfe49ed9700fe867db9f67d; no voters:
I20260924 00:09:34.845127 2032 raft_consensus.cc:2781] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:34.845218 2032 raft_consensus.cc:686] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 LEADER]: Becoming Leader. State: Replica: 2b730ad1cdfe49ed9700fe867db9f67d, State: Running, Role: LEADER
I20260924 00:09:34.845273 2032 consensus_queue.cc:237] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } }
I20260924 00:09:34.845530 2024 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: TxnStatusTablet state changed. Reason: New leader 2b730ad1cdfe49ed9700fe867db9f67d. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.845609 2024 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:34.845803 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:09:34.845822 1506 catalog_manager.cc:5495] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:34.845914 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:09:34.853380 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.850579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34431 (local address 127.0.0.1:35762)
0924 00:09:34.850641 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.850654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.850713 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.850862 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.850863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.850864 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:34.850904 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.850906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.851775 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.851776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.852408 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.852410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.852892 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.852893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.852991 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.853005 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.853217 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.853226 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.853285 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.853324 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 00:09:34.853385 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.850614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.0.98.2:34431)
0924 00:09:34.850659 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.850661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.850690 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:09:34.850778 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.850779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.850790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.850859 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:34.850942 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.851699 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.852488 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.852792 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.852867 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.853107 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.853161 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.853216 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.853293 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.853314 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:34.854383 1945 raft_consensus.cc:1261] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Refusing update from remote peer 2b730ad1cdfe49ed9700fe867db9f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.854382 1665 raft_consensus.cc:1261] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Refusing update from remote peer 2b730ad1cdfe49ed9700fe867db9f67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.854573 2024 consensus_queue.cc:1035] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.854660 2032 consensus_queue.cc:1035] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.855273 2018 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: TxnStatusTablet state changed. Reason: New leader 2b730ad1cdfe49ed9700fe867db9f67d. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.855309 2021 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: TxnStatusTablet state changed. Reason: New leader 2b730ad1cdfe49ed9700fe867db9f67d. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.855365 2018 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.855388 2021 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.856206 2022 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.856293 2022 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:34.856714 2018 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.856825 2018 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 23dd5c1cab824af996193d8d8762e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.860230 2022 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.860322 2022 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 2b730ad1cdfe49ed9700fe867db9f67d: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:34.860442 2021 tablet_replica.cc:439] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c900097e464c4ca2014e94ffe0c09f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36399 } } peers { permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 } } peers { permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 } } }
I20260924 00:09:34.860519 2021 tablet_replica.cc:442] T a781b423afed442a99e785cf45b1903b P 41c900097e464c4ca2014e94ffe0c09f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:34.861243 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.858311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.98.1:36399)
0924 00:09:34.858349 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.858351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.858374 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:09:34.858412 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.858412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.858428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.858480 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:34.858606 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.859055 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.860066 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.860356 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.860421 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.860628 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.860797 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.860891 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.861163 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.861184 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:09:34.866226 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.863623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36349 (local address 127.0.98.2:34431)
0924 00:09:34.863668 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.863670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.863779 (+ 109us) server_negotiation.cc:407] Connection header received
0924 00:09:34.863840 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.863840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.863849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.863890 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:34.864020 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.864457 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.865527 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.865706 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.865768 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.865969 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.865999 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.866055 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.866141 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.866159 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:09:34.866273 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.863538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34431 (local address 127.0.98.1:36349)
0924 00:09:34.863707 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.863724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.863783 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.863901 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.863902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.863904 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:34.863967 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.863968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.864525 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.864526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.865440 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.865442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.865780 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.865781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.865846 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.865861 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.866080 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.866082 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.866131 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.866173 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260924 00:09:34.868403 1945 raft_consensus.cc:1261] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Refusing update from remote peer 41c900097e464c4ca2014e94ffe0c09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.868494 1805 raft_consensus.cc:1261] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Refusing update from remote peer 41c900097e464c4ca2014e94ffe0c09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:34.868965 2045 consensus_queue.cc:1035] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23dd5c1cab824af996193d8d8762e7f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.869091 2021 consensus_queue.cc:1035] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b730ad1cdfe49ed9700fe867db9f67d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:34.878232 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.875131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36399 (local address 127.0.98.2:39635)
0924 00:09:34.875378 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:34.875389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:34.875462 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:34.875613 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:34.875613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:34.875615 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:34.875666 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.875668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.876529 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.876529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.877337 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:34.877340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.877822 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.877823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:34.877877 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.877899 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.878058 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.878060 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:34.878110 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:09:34.878155 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260924 00:09:34.878225 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.875194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39635 (local address 127.0.98.1:36399)
0924 00:09:34.875260 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.875262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.875436 (+ 174us) server_negotiation.cc:407] Connection header received
0924 00:09:34.875543 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.875544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.875556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.875609 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:34.875730 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.876449 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.877431 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.877727 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.877811 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.877979 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:34.878013 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:34.878059 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.878115 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.878132 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:09:34.887463 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:34.887579 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:34.888195 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3db65268371f48849e614ca134dd0a4e: Sending DeleteTablet for 3 replicas of tablet 4fb1fc812093445cbb55accfacba597a
I20260924 00:09:34.888453 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb1fc812093445cbb55accfacba597a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37424
I20260924 00:09:34.888511 1919 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb1fc812093445cbb55accfacba597a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47760
I20260924 00:09:34.888631 2039 tablet_replica.cc:330] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: stopping tablet replica
I20260924 00:09:34.888690 2039 raft_consensus.cc:2229] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:34.888736 2039 raft_consensus.cc:2258] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.888751 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fb1fc812093445cbb55accfacba597a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35744
I20260924 00:09:34.888557 2037 tablet_replica.cc:330] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: stopping tablet replica
I20260924 00:09:34.888846 2038 tablet_replica.cc:330] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: stopping tablet replica
I20260924 00:09:34.888888 2037 raft_consensus.cc:2229] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:34.888912 2038 raft_consensus.cc:2229] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:34.888947 2038 raft_consensus.cc:2258] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.888962 2037 raft_consensus.cc:2258] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:34.889289 2038 ts_tablet_manager.cc:1905] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.889312 2039 ts_tablet_manager.cc:1905] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.890985 2037 ts_tablet_manager.cc:1905] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:34.890995 2039 ts_tablet_manager.cc:1918] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:34.891003 2038 ts_tablet_manager.cc:1918] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:34.891043 2039 log.cc:1196] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/4fb1fc812093445cbb55accfacba597a
I20260924 00:09:34.891055 2038 log.cc:1196] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/4fb1fc812093445cbb55accfacba597a
I20260924 00:09:34.891292 2039 ts_tablet_manager.cc:1939] T 4fb1fc812093445cbb55accfacba597a P 23dd5c1cab824af996193d8d8762e7f5: Deleting consensus metadata
I20260924 00:09:34.891510 2038 ts_tablet_manager.cc:1939] T 4fb1fc812093445cbb55accfacba597a P 2b730ad1cdfe49ed9700fe867db9f67d: Deleting consensus metadata
I20260924 00:09:34.891602 1493 catalog_manager.cc:4841] TS 23dd5c1cab824af996193d8d8762e7f5 (127.0.98.3:39005): tablet 4fb1fc812093445cbb55accfacba597a (table test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]) successfully deleted
I20260924 00:09:34.891891 1495 catalog_manager.cc:4841] TS 2b730ad1cdfe49ed9700fe867db9f67d (127.0.98.2:34431): tablet 4fb1fc812093445cbb55accfacba597a (table test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]) successfully deleted
I20260924 00:09:34.892584 2037 ts_tablet_manager.cc:1918] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:34.892644 2037 log.cc:1196] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/4fb1fc812093445cbb55accfacba597a
I20260924 00:09:34.892848 2037 ts_tablet_manager.cc:1939] T 4fb1fc812093445cbb55accfacba597a P 41c900097e464c4ca2014e94ffe0c09f: Deleting consensus metadata
I20260924 00:09:34.893180 1492 catalog_manager.cc:4841] TS 41c900097e464c4ca2014e94ffe0c09f (127.0.98.1:36399): tablet 4fb1fc812093445cbb55accfacba597a (table test-table [id=5a21ef4a7ef54a97a2e316d22cea962f]) successfully deleted
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208574, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:34.905784 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.889602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.0.98.1:36399)
0924 00:09:34.889640 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.889642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.895425 (+ 5783us) server_negotiation.cc:407] Connection header received
0924 00:09:34.895479 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.895480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.895495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.895561 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.896300 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.897024 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.898151 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.898453 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.898493 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.902226 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.902234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.902235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.902242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.904346 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.904760 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.904763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.904764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.904835 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.905030 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.905031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.905031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.905202 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.905362 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.905579 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.905661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 00:09:34.906302 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37434
I20260924 00:09:34.914961 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:34.907354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.98.62:35483)
0924 00:09:34.907396 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:34.907398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:34.907599 (+ 201us) server_negotiation.cc:407] Connection header received
0924 00:09:34.907663 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:34.907664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:34.907679 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:34.907733 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:34.908395 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.909235 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.910227 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:34.910548 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:34.910630 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:34.912051 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:34.912053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:34.912053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:34.912060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:34.913735 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.914129 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.914131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.914131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.914193 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:34.914401 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:34.914402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:34.914402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:34.914552 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:34.914668 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:34.914767 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 00:09:34.914857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 00:09:34.915532 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50216
I20260924 00:09:34.916173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1580
I20260924 00:09:34.921511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1720
I20260924 00:09:34.927626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1860
I20260924 00:09:34.932039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 1476
2026-09-24T00:09:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2509 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[2079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[2079](info): set up 2 sockets
Sep 24 00:09:34 dist-test-slave-3q8k krb5kdc[2079](info): commencing operation
krb5kdc: starting...
W20260924 00:09:36.022941 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:36.144479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33423
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:36.234026 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:36.234247 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:36.234272 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:36.235606 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:36.235648 2092 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:09:36.235666 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:36.235682 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:36.235695 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:36.235708 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:36.237277 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33423
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:36.237478 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:36.237699 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:36.243136 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.243177 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.243199 2092 server_base.cc:1034] running on GCE node
W20260924 00:09:36.243110 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.243686 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:36.243949 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:36.245090 2092 hybrid_clock.cc:648] HybridClock initialized: now 1790208576245071 us; error 32 us; skew 500 ppm
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:36.247685 2092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:36.248065 2092 webserver.cc:460] Webserver started at http://127.0.98.62:34681/ using document root <none> and password file <none>
I20260924 00:09:36.248312 2092 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:36.248358 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:36.248500 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:36.249995 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "7e57af915a5c41efb1b789417cf9470b"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "541cf146004bf1fbfc9f3d8cae1be030"
server_key_iv: "852fd47033133089b18f0c0236261f33"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.250357 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e57af915a5c41efb1b789417cf9470b"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "541cf146004bf1fbfc9f3d8cae1be030"
server_key_iv: "852fd47033133089b18f0c0236261f33"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.251809 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:36.252552 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.252739 2092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:36.252837 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7e57af915a5c41efb1b789417cf9470b"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "541cf146004bf1fbfc9f3d8cae1be030"
server_key_iv: "852fd47033133089b18f0c0236261f33"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.252918 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:36.266776 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:36.269013 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:36.269280 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:36.273980 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33423
I20260924 00:09:36.274031 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33423 every 8 connection(s)
I20260924 00:09:36.274394 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:36.275049 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:36.277297 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Bootstrap starting.
I20260924 00:09:36.277899 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:36.278184 2170 log.cc:824] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:36.278851 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: No bootstrap required, opened a new log
I20260924 00:09:36.279827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2092
I20260924 00:09:36.279922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:36.280089 392 external_mini_cluster.cc:1428] Setting key 7e36db6c2a61dbd1d6b517a68431ca1a
I20260924 00:09:36.281438 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } }
I20260924 00:09:36.281613 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:36.281657 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e57af915a5c41efb1b789417cf9470b, State: Initialized, Role: FOLLOWER
I20260924 00:09:36.281806 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } }
I20260924 00:09:36.281896 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:36.281940 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:36.282011 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:36.282567 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } }
I20260924 00:09:36.282684 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e57af915a5c41efb1b789417cf9470b; no voters:
I20260924 00:09:36.282881 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:36.282922 2175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:36.283048 2175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [term 1 LEADER]: Becoming Leader. State: Replica: 7e57af915a5c41efb1b789417cf9470b, State: Running, Role: LEADER
I20260924 00:09:36.283136 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:36.283128 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } }
I20260924 00:09:36.284328 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e57af915a5c41efb1b789417cf9470b. Latest consensus state: current_term: 1 leader_uuid: "7e57af915a5c41efb1b789417cf9470b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } } }
I20260924 00:09:36.284418 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:36.284622 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:36.284816 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e57af915a5c41efb1b789417cf9470b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e57af915a5c41efb1b789417cf9470b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33423 } } }
I20260924 00:09:36.285281 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:36.285660 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:36.287499 2190 catalog_manager.cc:1347] Generated new cluster ID: 0ae0eb79c4c743879c4d0017842953e8
I20260924 00:09:36.287554 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:36.292855 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.280621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.62:33423)
0924 00:09:36.280805 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.280813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.280836 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:09:36.281435 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.281442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.281505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.281700 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.282635 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.284084 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.285125 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.285720 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.285844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.289057 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.289061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.289062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.289075 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.291119 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.291575 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.291578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.291579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.291707 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.291926 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.291926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.291927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.292239 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.292403 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.292500 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.292646 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260924 00:09:36.299306 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:36.300005 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:36.304852 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Generated new TSK 0
I20260924 00:09:36.305028 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:36.332410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33423
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:36.412966 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:36.413192 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:36.413216 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:36.414373 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:36.414415 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:36.414448 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:09:36.414460 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:36.414521 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:36.416039 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:36.416307 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:36.416538 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:09:36.421689 2196 server_base.cc:1034] running on GCE node
W20260924 00:09:36.421638 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.421622 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.421610 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.422034 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:36.422278 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:36.423437 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790208576423431 us; error 38 us; skew 500 ppm
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:36.426056 2196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:36.426436 2196 webserver.cc:460] Webserver started at http://127.0.98.1:42015/ using document root <none> and password file <none>
I20260924 00:09:36.426625 2196 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:36.426666 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:36.426750 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:36.427855 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "365beb42c86d434c95c3142a85000cfb"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "8cab12cbfe598f9f4160da5c8ce0ddf8"
server_key_iv: "ccb4efd237f3b9ddb4a0cb35cdeb5779"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.428149 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "365beb42c86d434c95c3142a85000cfb"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "8cab12cbfe598f9f4160da5c8ce0ddf8"
server_key_iv: "ccb4efd237f3b9ddb4a0cb35cdeb5779"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.429369 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:36.430075 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.430259 2196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:36.430351 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "365beb42c86d434c95c3142a85000cfb"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "8cab12cbfe598f9f4160da5c8ce0ddf8"
server_key_iv: "ccb4efd237f3b9ddb4a0cb35cdeb5779"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.430406 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:36.444223 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:36.446113 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:36.446226 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:36.446692 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:36.446739 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.446764 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:36.446823 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:36.453909 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39475
I20260924 00:09:36.453967 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39475 every 8 connection(s)
I20260924 00:09:36.454337 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:36.457329 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.447211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32867 (local address 127.0.98.62:33423)
0924 00:09:36.447308 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.447311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.447530 (+ 219us) server_negotiation.cc:407] Connection header received
0924 00:09:36.448022 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.448023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.448041 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.448127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.449005 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.449861 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.451444 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.451678 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.451762 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.454337 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.454339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.454339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.454349 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.455997 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.456395 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.456396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.456396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.456465 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.456723 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.456724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.456724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.456839 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.456892 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.457114 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.457216 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:09:36.457381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2196
I20260924 00:09:36.457372 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.447211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.1:32867)
0924 00:09:36.447390 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.447429 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.447921 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.448226 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.448231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.448608 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.448893 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.448901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.449969 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.449970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.451368 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.451371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.451780 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.451781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.451915 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.452446 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.452454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.454229 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.456091 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.456099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.456101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.456326 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.456530 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.456530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.456531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.456665 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.456898 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.456902 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.457012 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.457097 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.457217 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260924 00:09:36.457465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:36.457626 392 external_mini_cluster.cc:1428] Setting key a68138e1d473a5b56b4af076a6caf7d2
I20260924 00:09:36.457983 2327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33423
I20260924 00:09:36.458061 2327 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:36.458199 2327 heartbeater.cc:511] Master 127.0.98.62:33423 requested a full tablet report, sending...
I20260924 00:09:36.458734 2121 ts_manager.cc:194] Registered new tserver with Master: 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475)
I20260924 00:09:36.459602 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32867
I20260924 00:09:36.464962 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.458999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55055 (local address 127.0.98.62:33423)
0924 00:09:36.459035 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.459037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.459410 (+ 373us) server_negotiation.cc:407] Connection header received
0924 00:09:36.459467 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.459468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.459479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.459536 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.459977 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.460446 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.461189 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.461381 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.461470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.462443 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.462445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.462445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.462451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.463817 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.464167 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.464168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.464169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.464220 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.464439 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.464440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.464440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.464556 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.464632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.464792 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.464871 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:09:36.464993 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.458940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.1:55055)
0924 00:09:36.459351 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.459361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.459419 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.459583 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.459584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.459799 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.459874 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.459876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.460543 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.460544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.461067 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.461068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.461470 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.461471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.461545 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.462061 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.462065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.462352 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.463914 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.463915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.463915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.464080 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.464313 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.464314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.464314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.464373 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.464625 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.464626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.464671 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.464754 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.464872 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:36.482218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33423
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:36.565259 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:36.565474 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:36.565500 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:36.566674 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:36.566717 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:36.566740 2336 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:09:36.566751 2336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:36.566768 2336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:36.568216 2336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:36.568408 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:36.568593 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:36.573580 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.573601 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.573570 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.573886 2336 server_base.cc:1034] running on GCE node
I20260924 00:09:36.574147 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:36.574414 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:36.575582 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790208576575579 us; error 39 us; skew 500 ppm
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:36.578033 2336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:36.578441 2336 webserver.cc:460] Webserver started at http://127.0.98.2:34503/ using document root <none> and password file <none>
I20260924 00:09:36.578620 2336 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:36.578660 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:36.578747 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:36.579824 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "af791e73e3aa4f518fe2f611620e8882"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "993b1836742f79ef5029012a665cca49"
server_key_iv: "eb50d992fba129edbfad56246b0c8346"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.580127 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af791e73e3aa4f518fe2f611620e8882"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "993b1836742f79ef5029012a665cca49"
server_key_iv: "eb50d992fba129edbfad56246b0c8346"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.581461 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:36.582140 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.582309 2336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:36.582398 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "af791e73e3aa4f518fe2f611620e8882"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "993b1836742f79ef5029012a665cca49"
server_key_iv: "eb50d992fba129edbfad56246b0c8346"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.582492 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:36.607412 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:36.609297 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:36.609409 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:36.609862 2336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:36.609897 2336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.609925 2336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:36.609983 2336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.617092 2336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42485
I20260924 00:09:36.617190 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42485 every 8 connection(s)
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:36.617611 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:36.620749 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.610440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42771 (local address 127.0.98.62:33423)
0924 00:09:36.610524 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.610528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.610719 (+ 191us) server_negotiation.cc:407] Connection header received
0924 00:09:36.611249 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.611251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.611263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.611353 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.612233 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.613071 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.614673 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.614893 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.614997 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.617978 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.617980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.617981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.617989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.619565 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.619928 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.619929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.619929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.619978 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.620235 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.620235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.620236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.620336 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.620386 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.620625 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.620684 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:09:36.620929 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.610392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.2:42771)
0924 00:09:36.610585 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.610625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.611125 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.611422 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.611426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.611802 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.612116 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.612124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.613176 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.613177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.614582 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.614585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.615016 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.615017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.615173 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.615899 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.615909 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.617884 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.619636 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.619640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.619642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.619861 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.620035 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.620035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.620036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.620131 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.620393 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.620398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.620514 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.620600 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.620722 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":121,"threads_started":1}
I20260924 00:09:36.621371 2467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33423
I20260924 00:09:36.621451 2467 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:36.621649 2467 heartbeater.cc:511] Master 127.0.98.62:33423 requested a full tablet report, sending...
I20260924 00:09:36.622046 2122 ts_manager.cc:194] Registered new tserver with Master: af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485)
I20260924 00:09:36.622395 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42771
I20260924 00:09:36.627470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2336
I20260924 00:09:36.627545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:36.627732 392 external_mini_cluster.cc:1428] Setting key b311321c5e0553c57a032b004c76e063
I20260924 00:09:36.627982 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.622517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56751 (local address 127.0.98.62:33423)
0924 00:09:36.622554 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.622556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.622729 (+ 173us) server_negotiation.cc:407] Connection header received
0924 00:09:36.622800 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.622801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.622811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.622866 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.623175 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.623675 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.624378 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.624580 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.624662 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.625614 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.625616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.625616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.625622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.626744 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.627087 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.627088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.627088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.627140 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.627399 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.627400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.627400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.627591 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.627651 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.627825 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.627891 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:09:36.627997 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.622507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.2:56751)
0924 00:09:36.622647 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.622657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.622742 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.622871 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.622872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.623050 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.623098 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.623099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.623770 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.623771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.624280 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.624282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.624664 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.624665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.624749 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.625219 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.625224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.625507 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.626828 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.626829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.626830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.626999 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.627243 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.627244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.627244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.627322 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.627654 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.627655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.627707 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.627787 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.627892 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:36.654476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33423
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:36.736935 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:36.737152 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:36.737185 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:36.738363 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:36.738407 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:36.738428 2476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:09:36.738466 2476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:09:36.738504 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:36.739996 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:36.740262 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:36.740502 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:36.745488 2481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:36.745582 2482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.745700 2476 server_base.cc:1034] running on GCE node
W20260924 00:09:36.745740 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:36.745944 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:36.746201 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:36.747377 2476 hybrid_clock.cc:648] HybridClock initialized: now 1790208576747364 us; error 41 us; skew 500 ppm
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:36.749859 2476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:36.750372 2476 webserver.cc:460] Webserver started at http://127.0.98.3:42791/ using document root <none> and password file <none>
I20260924 00:09:36.750605 2476 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:36.750656 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:36.750823 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:36.751955 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "3a57c8d7222fc3085c15ad141ebc82ba"
server_key_iv: "cb3ee7b22612a7301a453847f36eeacc"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.752308 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "3a57c8d7222fc3085c15ad141ebc82ba"
server_key_iv: "cb3ee7b22612a7301a453847f36eeacc"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.753648 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:36.754371 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.754575 2476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:36.754652 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
format_stamp: "Formatted at 2026-09-24 00:09:36 on dist-test-slave-3q8k"
server_key: "3a57c8d7222fc3085c15ad141ebc82ba"
server_key_iv: "cb3ee7b22612a7301a453847f36eeacc"
server_key_version: "encryptionkey@0"
I20260924 00:09:36.754703 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:36.782235 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:36.784698 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:36.784861 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:36.785435 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:36.785475 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.785498 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:36.785511 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:36.792673 2476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44913
I20260924 00:09:36.792739 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44913 every 8 connection(s)
I20260924 00:09:36.793040 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:36.796581 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.785913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42865 (local address 127.0.98.62:33423)
0924 00:09:36.785957 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.785959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.786205 (+ 246us) server_negotiation.cc:407] Connection header received
0924 00:09:36.786807 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.786808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.786822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.786879 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.787949 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.788806 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.790588 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.790858 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.790919 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.793849 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.793851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.793852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.793859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.795396 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.795797 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.795798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.795798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.795849 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.796062 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.796063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.796063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.796166 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.796217 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.796456 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.796517 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:09:36.796761 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.785918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.3:42865)
0924 00:09:36.786086 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.786121 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.786676 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.787006 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.787011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.787496 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.787824 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.787832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.788928 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.788929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.790480 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.790483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.790953 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.790954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.791138 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.791832 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.791843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.793745 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.795471 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.795477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.795486 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.795716 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.795909 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.795909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.795910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.796007 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.796223 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.796227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.796344 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.796429 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.796561 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260924 00:09:36.797271 2607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33423
I20260924 00:09:36.797363 2607 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:36.797546 2607 heartbeater.cc:511] Master 127.0.98.62:33423 requested a full tablet report, sending...
I20260924 00:09:36.797928 2121 ts_manager.cc:194] Registered new tserver with Master: 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:36.798432 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42865
I20260924 00:09:36.799536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2476
I20260924 00:09:36.799607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:36.799829 392 external_mini_cluster.cc:1428] Setting key 107de2fd0805e922763f873e3496a890
I20260924 00:09:36.800725 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:09:36.805050 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.798520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51459 (local address 127.0.98.62:33423)
0924 00:09:36.798562 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.798564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.798693 (+ 129us) server_negotiation.cc:407] Connection header received
0924 00:09:36.798781 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.798782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.798792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.798842 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.799137 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.799734 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.800697 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.800969 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.801051 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.802006 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.802008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.802008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.802014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.803773 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.804210 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.804211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.804212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.804267 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.804482 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.804483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.804483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.804590 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.804655 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.804913 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.804990 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:36.805166 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.798442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.98.3:51459)
0924 00:09:36.798629 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.798640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.798705 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.798849 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.798850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.799012 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:36.799061 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.799062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.799830 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.799831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.800623 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.800625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.801054 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.801055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.801122 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.801598 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:36.801603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:36.801898 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:36.803877 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.803878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.803878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.804105 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.804347 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:36.804348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:36.804349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:36.804420 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:36.804743 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:36.804744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:36.804791 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.804868 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.805003 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":136,"threads_started":1}
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:09:36 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:36.824699 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.817670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.98.62:33423)
0924 00:09:36.817718 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.817720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.817728 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:36.817795 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.817795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.817804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.817853 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:36.818450 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.818947 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.819796 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.819975 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.820028 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.822453 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:36.822455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:36.822455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:36.822462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:36.823804 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.824109 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.824110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.824110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.824158 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:36.824320 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:36.824321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:36.824321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:36.824417 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:36.824490 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.824576 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.824633 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:36.826327 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:36.827973 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:36.834501 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39475 (local address 127.0.0.1:47670)
0924 00:09:36.830626 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.830636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.830691 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.830994 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.830995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.830997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:36.831055 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.831057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.832003 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.832004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.833226 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.833229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834163 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.834244 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834259 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.834336 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.834412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260924 00:09:36.834587 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.0.98.3:44913)
0924 00:09:36.830624 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.830630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.830640 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:36.830680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.830682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.830698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.830792 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:36.831011 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.832189 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.833530 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834233 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834296 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834343 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.834456 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.834492 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":78,"threads_started":1}
I20260924 00:09:36.834609 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44913 (local address 127.0.0.1:42224)
0924 00:09:36.830359 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.830399 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.830461 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.830832 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.830836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.830838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:36.830921 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.830923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.832276 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.832277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.833441 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.833444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834306 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.834382 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834386 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.834441 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.834481 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":48,"thread_start_us":57,"threads_started":1}
I20260924 00:09:36.834928 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.0.98.2:42485)
0924 00:09:36.830624 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.830630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.830819 (+ 189us) server_negotiation.cc:407] Connection header received
0924 00:09:36.830981 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.830984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.830999 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.831071 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:36.831267 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.832317 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.833648 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834326 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834384 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834436 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.834798 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.834836 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260924 00:09:36.835139 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.0.1:34264)
0924 00:09:36.830746 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:36.830752 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:36.830846 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:36.831132 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:36.831133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:36.831134 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:36.831165 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.831166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.832411 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.832412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.833588 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:36.833590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834471 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:36.834530 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834534 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:36.835017 (+ 483us) client_negotiation.cc:240] Negotiation successful
0924 00:09:36.835076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260924 00:09:36.835425 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:36.830450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.0.98.1:39475)
0924 00:09:36.830719 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:36.830723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:36.830735 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:09:36.830770 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:36.830771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:36.830785 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:36.830921 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:36.831158 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.831911 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.833309 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:36.834076 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:36.834152 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:36.834212 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:36.834959 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 00:09:36.835123 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":74,"threads_started":1}
I20260924 00:09:36.835453 2541 tablet_service.cc:1467] Processing CreateTablet for tablet abeb7306a46240378adc1a2fd4dbb87a (DEFAULT_TABLE table=test-table [id=b522fbe23db845ad88839401b6178f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:36.835870 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abeb7306a46240378adc1a2fd4dbb87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:36.835821 2401 tablet_service.cc:1467] Processing CreateTablet for tablet abeb7306a46240378adc1a2fd4dbb87a (DEFAULT_TABLE table=test-table [id=b522fbe23db845ad88839401b6178f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:36.836193 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abeb7306a46240378adc1a2fd4dbb87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:36.836112 2261 tablet_service.cc:1467] Processing CreateTablet for tablet abeb7306a46240378adc1a2fd4dbb87a (DEFAULT_TABLE table=test-table [id=b522fbe23db845ad88839401b6178f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:36.836411 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abeb7306a46240378adc1a2fd4dbb87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:36.839144 2630 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Bootstrap starting.
I20260924 00:09:36.839532 2631 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Bootstrap starting.
I20260924 00:09:36.839951 2630 tablet_bootstrap.cc:654] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:36.840031 2632 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Bootstrap starting.
I20260924 00:09:36.840371 2630 log.cc:824] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:36.840423 2631 tablet_bootstrap.cc:654] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:36.840687 2632 tablet_bootstrap.cc:654] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:36.840777 2631 log.cc:824] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:36.841048 2632 log.cc:824] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:36.841209 2630 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: No bootstrap required, opened a new log
I20260924 00:09:36.841306 2630 ts_tablet_manager.cc:1397] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:36.841614 2631 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: No bootstrap required, opened a new log
I20260924 00:09:36.841681 2631 ts_tablet_manager.cc:1397] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:36.843358 2630 raft_consensus.cc:348] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.843539 2630 raft_consensus.cc:374] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:36.843586 2630 raft_consensus.cc:729] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Initialized, Role: FOLLOWER
I20260924 00:09:36.843713 2630 consensus_queue.cc:260] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.843668 2631 raft_consensus.cc:348] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.843817 2631 raft_consensus.cc:374] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:36.843864 2631 raft_consensus.cc:729] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Initialized, Role: FOLLOWER
I20260924 00:09:36.843921 2632 tablet_bootstrap.cc:492] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: No bootstrap required, opened a new log
I20260924 00:09:36.843990 2630 ts_tablet_manager.cc:1428] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:36.843993 2632 ts_tablet_manager.cc:1397] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:09:36.843977 2631 consensus_queue.cc:260] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.844115 2467 heartbeater.cc:503] Master 127.0.98.62:33423 was elected leader, sending a full tablet report...
I20260924 00:09:36.844285 2607 heartbeater.cc:503] Master 127.0.98.62:33423 was elected leader, sending a full tablet report...
I20260924 00:09:36.844234 2631 ts_tablet_manager.cc:1428] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:36.846032 2632 raft_consensus.cc:348] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.846184 2632 raft_consensus.cc:374] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:36.846228 2632 raft_consensus.cc:729] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Initialized, Role: FOLLOWER
I20260924 00:09:36.846347 2632 consensus_queue.cc:260] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:36.846613 2632 ts_tablet_manager.cc:1428] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:36.846676 2327 heartbeater.cc:503] Master 127.0.98.62:33423 was elected leader, sending a full tablet report...
W20260924 00:09:36.868327 2468 tablet.cc:2369] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:36.955392 2328 tablet.cc:2369] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:37.043668 2608 tablet.cc:2369] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:37.144182 2637 raft_consensus.cc:482] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:37.144327 2637 raft_consensus.cc:504] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:37.144630 2637 leader_election.cc:288] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485)
I20260924 00:09:37.147676 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.144929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37535 (local address 127.0.98.1:39475)
0924 00:09:37.144975 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.144977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.145020 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:09:37.145046 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.145046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.145055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.145099 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:37.145290 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.145875 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.146847 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.147325 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.147380 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.147389 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.147554 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.147597 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:37.147679 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.144851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39475 (local address 127.0.98.3:37535)
0924 00:09:37.144898 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.144910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.144962 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.145133 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.145134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.145136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:37.145215 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.145217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.145958 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.145959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.146752 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.146754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.147398 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.147399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.147477 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.147481 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.147534 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.147576 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 00:09:37.147764 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.144911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58757 (local address 127.0.98.2:42485)
0924 00:09:37.144963 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.144965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.145316 (+ 351us) server_negotiation.cc:407] Connection header received
0924 00:09:37.145351 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.145352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.145361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.145402 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:37.145551 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.146055 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.146951 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.147399 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.147476 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.147486 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.147653 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.147692 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:09:37.147826 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.144851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.98.3:58757)
0924 00:09:37.145209 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.145218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.145299 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.145423 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.145424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.145425 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:37.145488 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.145490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.146136 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.146137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.146860 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.146862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.147483 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.147484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.147574 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.147578 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.147635 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.147670 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":158,"thread_start_us":112,"threads_started":1}
I20260924 00:09:37.148088 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abeb7306a46240378adc1a2fd4dbb87a" candidate_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af791e73e3aa4f518fe2f611620e8882" is_pre_election: true
I20260924 00:09:37.148088 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abeb7306a46240378adc1a2fd4dbb87a" candidate_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb" is_pre_election: true
I20260924 00:09:37.148247 2281 raft_consensus.cc:2454] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a387c5b443c47eea5a2b0fe6a6d43e4 in term 0.
I20260924 00:09:37.148247 2421 raft_consensus.cc:2454] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a387c5b443c47eea5a2b0fe6a6d43e4 in term 0.
I20260924 00:09:37.148487 2495 leader_election.cc:302] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365beb42c86d434c95c3142a85000cfb, 6a387c5b443c47eea5a2b0fe6a6d43e4; no voters:
I20260924 00:09:37.148649 2637 raft_consensus.cc:2781] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:37.148725 2637 raft_consensus.cc:482] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:37.148753 2637 raft_consensus.cc:3037] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.149719 2637 raft_consensus.cc:504] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:37.149881 2637 leader_election.cc:288] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [CANDIDATE]: Term 1 election: Requested vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485)
I20260924 00:09:37.150054 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abeb7306a46240378adc1a2fd4dbb87a" candidate_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb"
I20260924 00:09:37.150061 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abeb7306a46240378adc1a2fd4dbb87a" candidate_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af791e73e3aa4f518fe2f611620e8882"
I20260924 00:09:37.150135 2421 raft_consensus.cc:3037] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.150135 2281 raft_consensus.cc:3037] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.150966 2421 raft_consensus.cc:2454] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a387c5b443c47eea5a2b0fe6a6d43e4 in term 1.
I20260924 00:09:37.150971 2281 raft_consensus.cc:2454] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a387c5b443c47eea5a2b0fe6a6d43e4 in term 1.
I20260924 00:09:37.151211 2495 leader_election.cc:302] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365beb42c86d434c95c3142a85000cfb, 6a387c5b443c47eea5a2b0fe6a6d43e4; no voters:
I20260924 00:09:37.151335 2637 raft_consensus.cc:2781] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:37.151482 2637 raft_consensus.cc:686] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Running, Role: LEADER
I20260924 00:09:37.151607 2637 consensus_queue.cc:237] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } }
I20260924 00:09:37.152451 2122 catalog_manager.cc:5495] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:09:37 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:37.207584 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.200042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.0.98.62:33423)
0924 00:09:37.200117 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.200120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.200302 (+ 182us) server_negotiation.cc:407] Connection header received
0924 00:09:37.200392 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.200393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.200403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.200474 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:37.200965 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.201445 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.202176 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.202355 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.202428 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.204694 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:37.204696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:37.204696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:37.204702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:37.206285 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:37.206681 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:37.206682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:37.206683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:37.206750 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:37.206947 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:37.206948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:37.206948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:37.207061 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:37.207140 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.207304 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.207478 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":43}
I20260924 00:09:37.207584 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.199987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33423 (local address 127.0.0.1:58510)
0924 00:09:37.200217 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.200228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.200319 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.200478 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.200479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.200813 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:37.200868 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.200871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.201531 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.201533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.202063 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.202064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.202438 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.202438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.202498 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.203022 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:37.203026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:37.204571 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:37.206387 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:37.206395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:37.206399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:37.206603 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:37.206818 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:37.206819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:37.206819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:37.206878 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:37.207130 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:37.207138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:37.207185 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.207277 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.207430 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":6,"thread_start_us":93,"threads_started":1}
I20260924 00:09:37.209750 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58510:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:09:37.209996 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:37.211717 2401 tablet_service.cc:1467] Processing CreateTablet for tablet a5247b0b2c2b42f1907005c2061b8b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c4591f9a624f12a5e7ac57da2fa244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:37.211750 2541 tablet_service.cc:1467] Processing CreateTablet for tablet a5247b0b2c2b42f1907005c2061b8b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c4591f9a624f12a5e7ac57da2fa244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:37.211918 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5247b0b2c2b42f1907005c2061b8b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.211922 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5247b0b2c2b42f1907005c2061b8b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.213356 2630 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: Bootstrap starting.
I20260924 00:09:37.213414 2631 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Bootstrap starting.
I20260924 00:09:37.213483 2261 tablet_service.cc:1467] Processing CreateTablet for tablet a5247b0b2c2b42f1907005c2061b8b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35c4591f9a624f12a5e7ac57da2fa244]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:09:37.213647 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5247b0b2c2b42f1907005c2061b8b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.213833 2630 tablet_bootstrap.cc:654] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.213833 2631 tablet_bootstrap.cc:654] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.214571 2631 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: No bootstrap required, opened a new log
I20260924 00:09:37.214691 2631 ts_tablet_manager.cc:1397] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:37.214758 2630 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: No bootstrap required, opened a new log
I20260924 00:09:37.214814 2630 ts_tablet_manager.cc:1397] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:37.214859 2631 raft_consensus.cc:348] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.214983 2631 raft_consensus.cc:374] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.215008 2631 raft_consensus.cc:729] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.214979 2630 raft_consensus.cc:348] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.215065 2630 raft_consensus.cc:374] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.215076 2631 consensus_queue.cc:260] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.215197 2630 raft_consensus.cc:729] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.215245 2632 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: Bootstrap starting.
I20260924 00:09:37.215296 2630 consensus_queue.cc:260] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.215338 2637 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.215416 2630 ts_tablet_manager.cc:1428] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:37.215425 2637 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.215185 2631 ts_tablet_manager.cc:1428] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:37.215577 2636 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.215654 2636 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.215672 2632 tablet_bootstrap.cc:654] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.216535 2632 tablet_bootstrap.cc:492] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: No bootstrap required, opened a new log
I20260924 00:09:37.216599 2632 ts_tablet_manager.cc:1397] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:37.216753 2632 raft_consensus.cc:348] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.216832 2632 raft_consensus.cc:374] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.216856 2632 raft_consensus.cc:729] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.216905 2632 consensus_queue.cc:260] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.217020 2632 ts_tablet_manager.cc:1428] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:37.217094 2638 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.217221 2638 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.466375 2636 raft_consensus.cc:482] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:37.466480 2636 raft_consensus.cc:504] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.466894 2636 leader_election.cc:288] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:37.469954 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.467152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44913 (local address 127.0.98.2:43787)
0924 00:09:37.467307 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.467320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.467394 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.467624 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.467625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.467627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:37.467711 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.467713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.468412 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.468413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.469135 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.469136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.469742 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.469743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.469811 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.469814 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.469857 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.469887 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260924 00:09:37.470028 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.467251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43787 (local address 127.0.98.3:44913)
0924 00:09:37.467386 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.467388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.467398 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:37.467487 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.467488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.467496 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.467595 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:37.467792 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.468321 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.469245 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.469614 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.469706 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.469720 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.469884 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.469932 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260924 00:09:37.470281 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.467152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39475 (local address 127.0.98.2:53665)
0924 00:09:37.467308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.467320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.467392 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.467557 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.467558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.467561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:37.467642 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.467646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.468320 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.468320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.469137 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.469139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.470003 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.470004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.470095 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.470101 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.470155 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.470194 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":16,"thread_start_us":81,"threads_started":1}
I20260924 00:09:37.470279 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.467236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53665 (local address 127.0.98.1:39475)
0924 00:09:37.467309 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.467312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.467385 (+ 73us) server_negotiation.cc:407] Connection header received
0924 00:09:37.467456 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.467457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.467468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.467524 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:37.467727 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.468226 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.469234 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.469598 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.469653 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.469661 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.470165 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.470208 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:37.470449 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5247b0b2c2b42f1907005c2061b8b82" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" is_pre_election: true
I20260924 00:09:37.470564 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5247b0b2c2b42f1907005c2061b8b82" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb" is_pre_election: true
I20260924 00:09:37.470602 2561 raft_consensus.cc:2454] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:37.470643 2281 raft_consensus.cc:2454] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:37.470870 2354 leader_election.cc:302] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a387c5b443c47eea5a2b0fe6a6d43e4, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:37.471042 2636 raft_consensus.cc:2781] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:37.471098 2636 raft_consensus.cc:482] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:37.471127 2636 raft_consensus.cc:3037] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.471825 2636 raft_consensus.cc:504] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.471971 2636 leader_election.cc:288] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Requested vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:37.472210 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5247b0b2c2b42f1907005c2061b8b82" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
I20260924 00:09:37.472210 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5247b0b2c2b42f1907005c2061b8b82" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb"
I20260924 00:09:37.472298 2281 raft_consensus.cc:3037] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.472298 2561 raft_consensus.cc:3037] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.473061 2281 raft_consensus.cc:2454] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:37.473165 2561 raft_consensus.cc:2454] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:37.473294 2355 leader_election.cc:302] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365beb42c86d434c95c3142a85000cfb, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:37.473380 2636 raft_consensus.cc:2781] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:37.473512 2636 raft_consensus.cc:686] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [term 1 LEADER]: Becoming Leader. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Running, Role: LEADER
I20260924 00:09:37.473608 2636 consensus_queue.cc:237] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.474080 2654 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: TxnStatusTablet state changed. Reason: New leader af791e73e3aa4f518fe2f611620e8882. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.474171 2654 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:37.474417 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:09:37.474534 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:09:37.474462 2122 catalog_manager.cc:5495] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 reported cstate change: term changed from 0 to 1, leader changed from <none> to af791e73e3aa4f518fe2f611620e8882 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:37.500398 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.497649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.98.2:42485)
0924 00:09:37.497715 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.497717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.497727 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:37.497792 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.497793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.497818 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.497889 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.498033 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.498575 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.499550 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.499751 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.499836 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.500041 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.500146 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.500237 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.500284 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.500314 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:09:37.500398 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.497545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.0.1:34268)
0924 00:09:37.497633 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.497643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.497701 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.497895 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.497896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.497898 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:37.497951 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.497953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.498655 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.498657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.499462 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.499465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.499835 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.499836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.499904 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.499924 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.500213 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.500223 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.500270 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.500323 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 00:09:37.502327 2281 raft_consensus.cc:1261] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.502327 2561 raft_consensus.cc:1261] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.502666 2636 consensus_queue.cc:1035] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:37.502841 2654 consensus_queue.cc:1035] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:37.503901 2637 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: TxnStatusTablet state changed. Reason: New leader af791e73e3aa4f518fe2f611620e8882. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.504000 2637 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.503978 2638 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: TxnStatusTablet state changed. Reason: New leader af791e73e3aa4f518fe2f611620e8882. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.504072 2638 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.505192 2636 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.505283 2636 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:37.505425 2636 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.505438 2637 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.505492 2636 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P af791e73e3aa4f518fe2f611620e8882: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:09:37.505512 2637 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 6a387c5b443c47eea5a2b0fe6a6d43e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.505602 2659 mvcc.cc:205] Tried to move back new op lower bound from 7332694333446701056 to 7332694333332443136. Current Snapshot: MvccSnapshot[applied={T|T < 7332694333446701056}]
I20260924 00:09:37.505767 2638 tablet_replica.cc:439] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } } }
I20260924 00:09:37.505856 2638 tablet_replica.cc:442] T a5247b0b2c2b42f1907005c2061b8b82 P 365beb42c86d434c95c3142a85000cfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:09:37.506392 2660 mvcc.cc:205] Tried to move back new op lower bound from 7332694333446701056 to 7332694333332443136. Current Snapshot: MvccSnapshot[applied={T|T < 7332694333446701056}]
I20260924 00:09:37.506474 2661 mvcc.cc:205] Tried to move back new op lower bound from 7332694333446701056 to 7332694333332443136. Current Snapshot: MvccSnapshot[applied={T|T < 7332694333446701056}]
I20260924 00:09:37.510984 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.508402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.3:44913)
0924 00:09:37.508476 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.508478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.508487 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:37.508520 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.508521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.508541 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.508630 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.508719 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.509443 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.510193 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.510502 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.510575 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.510679 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.510799 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.510873 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.510891 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.510910 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:09:37.516139 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.513643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40803 (local address 127.0.98.2:42485)
0924 00:09:37.513685 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.513687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.513855 (+ 168us) server_negotiation.cc:407] Connection header received
0924 00:09:37.513925 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.513926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.513939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.513988 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.514130 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.514591 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.515437 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.515631 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.515707 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.515897 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.515933 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.515985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.516045 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.516064 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:09:37.516215 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.513570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.98.3:40803)
0924 00:09:37.513770 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.513780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.513872 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.513989 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.513990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.513992 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:37.514053 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.514055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.514678 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.514679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.515356 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.515359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.515705 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.515706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.515768 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.515783 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.515999 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.516001 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.516044 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.516085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260924 00:09:37.519155 2281 raft_consensus.cc:1261] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Refusing update from remote peer 6a387c5b443c47eea5a2b0fe6a6d43e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.519245 2421 raft_consensus.cc:1261] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Refusing update from remote peer 6a387c5b443c47eea5a2b0fe6a6d43e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.519503 2640 consensus_queue.cc:1035] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:37.519665 2637 consensus_queue.cc:1035] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:37.529348 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.525810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56461 (local address 127.0.98.3:44913)
0924 00:09:37.525863 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.525865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.526000 (+ 135us) server_negotiation.cc:407] Connection header received
0924 00:09:37.526061 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.526062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.526071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.526129 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.526280 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.526717 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.527980 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.528186 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.528239 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.528415 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.528440 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.528472 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.529241 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.529265 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:09:37.529563 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.525739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44913 (local address 127.0.98.2:56461)
0924 00:09:37.525935 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.525946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.526008 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.526140 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.526140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.526142 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:37.526219 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.526221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.526789 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.526790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.527912 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.527915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.528258 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.528259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.528324 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.528340 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.529149 (+ 809us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.529152 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.529435 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.529479 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260924 00:09:37.542280 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:37.542400 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:37.543193 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Sending DeleteTablet for 3 replicas of tablet abeb7306a46240378adc1a2fd4dbb87a
I20260924 00:09:37.543547 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet abeb7306a46240378adc1a2fd4dbb87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34264
I20260924 00:09:37.543607 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet abeb7306a46240378adc1a2fd4dbb87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260924 00:09:37.543862 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet abeb7306a46240378adc1a2fd4dbb87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 00:09:37.544067 2681 tablet_replica.cc:330] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: stopping tablet replica
I20260924 00:09:37.544179 2681 raft_consensus.cc:2229] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:37.544147 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:09:37.544235 2681 raft_consensus.cc:2258] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:09:37.544286 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:37.544477 2682 tablet_replica.cc:330] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: stopping tablet replica
I20260924 00:09:37.544562 2682 raft_consensus.cc:2229] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:37.544656 2682 raft_consensus.cc:2258] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:37.545074 2681 ts_tablet_manager.cc:1905] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:37.545696 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 (DEFAULT_TABLE table=test-table [id=023e85e2c2fe40f6b87b133f648c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:37.545809 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974fb070617f4121b8dfaf6b6614b8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.546875 2681 ts_tablet_manager.cc:1918] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:37.546939 2681 log.cc:1196] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/abeb7306a46240378adc1a2fd4dbb87a
I20260924 00:09:37.547118 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 (DEFAULT_TABLE table=test-table [id=023e85e2c2fe40f6b87b133f648c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:37.547207 2681 ts_tablet_manager.cc:1939] T abeb7306a46240378adc1a2fd4dbb87a P 365beb42c86d434c95c3142a85000cfb: Deleting consensus metadata
I20260924 00:09:37.547243 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974fb070617f4121b8dfaf6b6614b8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.547784 2682 ts_tablet_manager.cc:1905] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:37.548612 2632 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Bootstrap starting.
I20260924 00:09:37.548995 2632 tablet_bootstrap.cc:654] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.549316 2682 ts_tablet_manager.cc:1918] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:37.549389 2682 log.cc:1196] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/abeb7306a46240378adc1a2fd4dbb87a
I20260924 00:09:37.549321 2631 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Bootstrap starting.
I20260924 00:09:37.549666 2682 ts_tablet_manager.cc:1939] T abeb7306a46240378adc1a2fd4dbb87a P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting consensus metadata
I20260924 00:09:37.549815 2631 tablet_bootstrap.cc:654] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.550169 2111 catalog_manager.cc:4841] TS 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475): tablet abeb7306a46240378adc1a2fd4dbb87a (table test-table [id=b522fbe23db845ad88839401b6178f62]) successfully deleted
I20260924 00:09:37.550402 2110 catalog_manager.cc:4841] TS 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913): tablet abeb7306a46240378adc1a2fd4dbb87a (table test-table [id=b522fbe23db845ad88839401b6178f62]) successfully deleted
I20260924 00:09:37.550429 2631 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: No bootstrap required, opened a new log
I20260924 00:09:37.550469 2631 ts_tablet_manager.cc:1397] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:37.550611 2631 raft_consensus.cc:348] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.550697 2631 raft_consensus.cc:374] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.550719 2631 raft_consensus.cc:729] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.550760 2631 consensus_queue.cc:260] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.550798 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 (DEFAULT_TABLE table=test-table [id=023e85e2c2fe40f6b87b133f648c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:37.550844 2631 ts_tablet_manager.cc:1428] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:37.550913 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974fb070617f4121b8dfaf6b6614b8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:37.552299 2630 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Bootstrap starting.
I20260924 00:09:37.552687 2630 tablet_bootstrap.cc:654] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:37.553582 2630 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: No bootstrap required, opened a new log
I20260924 00:09:37.553642 2630 ts_tablet_manager.cc:1397] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:37.553779 2630 raft_consensus.cc:348] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.553845 2630 raft_consensus.cc:374] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.553864 2630 raft_consensus.cc:729] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.553906 2630 consensus_queue.cc:260] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.553995 2630 ts_tablet_manager.cc:1428] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:37.554085 2680 tablet_replica.cc:330] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: stopping tablet replica
I20260924 00:09:37.554231 2680 raft_consensus.cc:2229] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:37.554461 2680 raft_consensus.cc:2258] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:37.554888 2680 ts_tablet_manager.cc:1905] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:37.556728 2680 ts_tablet_manager.cc:1918] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:37.556806 2680 log.cc:1196] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/abeb7306a46240378adc1a2fd4dbb87a
I20260924 00:09:37.557070 2680 ts_tablet_manager.cc:1939] T abeb7306a46240378adc1a2fd4dbb87a P af791e73e3aa4f518fe2f611620e8882: Deleting consensus metadata
I20260924 00:09:37.557365 2632 tablet_bootstrap.cc:492] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: No bootstrap required, opened a new log
I20260924 00:09:37.557448 2632 ts_tablet_manager.cc:1397] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260924 00:09:37.557516 2109 catalog_manager.cc:4841] TS af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485): tablet abeb7306a46240378adc1a2fd4dbb87a (table test-table [id=b522fbe23db845ad88839401b6178f62]) successfully deleted
I20260924 00:09:37.557629 2632 raft_consensus.cc:348] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.557719 2632 raft_consensus.cc:374] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:37.557734 2632 raft_consensus.cc:729] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Initialized, Role: FOLLOWER
I20260924 00:09:37.557811 2632 consensus_queue.cc:260] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.557928 2632 ts_tablet_manager.cc:1428] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:37.761731 2636 raft_consensus.cc:482] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:37.761839 2636 raft_consensus.cc:504] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.762048 2636 leader_election.cc:288] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:37.762291 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974fb070617f4121b8dfaf6b6614b8d6" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" is_pre_election: true
I20260924 00:09:37.762327 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974fb070617f4121b8dfaf6b6614b8d6" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb" is_pre_election: true
I20260924 00:09:37.762404 2561 raft_consensus.cc:2454] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:37.762440 2281 raft_consensus.cc:2454] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:37.762565 2354 leader_election.cc:302] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a387c5b443c47eea5a2b0fe6a6d43e4, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:37.762656 2636 raft_consensus.cc:2781] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:37.762727 2636 raft_consensus.cc:482] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:37.762759 2636 raft_consensus.cc:3037] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.763530 2636 raft_consensus.cc:504] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.763669 2636 leader_election.cc:288] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Requested vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:37.763804 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974fb070617f4121b8dfaf6b6614b8d6" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
I20260924 00:09:37.763875 2561 raft_consensus.cc:3037] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.763852 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974fb070617f4121b8dfaf6b6614b8d6" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb"
I20260924 00:09:37.763954 2281 raft_consensus.cc:3037] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:37.764523 2561 raft_consensus.cc:2454] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:37.764680 2354 leader_election.cc:302] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a387c5b443c47eea5a2b0fe6a6d43e4, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:37.764680 2281 raft_consensus.cc:2454] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:37.764765 2636 raft_consensus.cc:2781] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:37.764818 2636 raft_consensus.cc:686] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 1 LEADER]: Becoming Leader. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Running, Role: LEADER
I20260924 00:09:37.764868 2636 consensus_queue.cc:237] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:37.765513 2122 catalog_manager.cc:5495] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 reported cstate change: term changed from 0 to 1, leader changed from <none> to af791e73e3aa4f518fe2f611620e8882 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:37.787467 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.784800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.98.2:42485)
0924 00:09:37.784858 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.784860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.784920 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:09:37.784992 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.784993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.785005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.785068 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.785206 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.785720 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.786551 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.786764 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.786827 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.787023 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.787053 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.787104 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.787189 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.787205 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:09:37.792062 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.789685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50355 (local address 127.0.98.2:42485)
0924 00:09:37.789741 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:37.789743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:37.789749 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:09:37.789799 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:37.789800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:37.789808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:37.789852 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:37.789944 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.790383 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.791105 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.791451 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.791531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.791767 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.791795 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.791848 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:37.791949 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:09:37.791969 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260924 00:09:37.792070 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:37.789617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.98.2:50355)
0924 00:09:37.789686 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:37.789698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:37.789741 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:37.789868 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:37.789869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:37.789870 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:37.789902 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.789903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.790469 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.790470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.791002 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:37.791004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:37.791562 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:37.791563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:37.791606 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:37.791612 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:37.791894 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:37.791896 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:37.791948 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:09:37.791991 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 00:09:37.794091 2561 raft_consensus.cc:1261] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.794191 2280 raft_consensus.cc:1261] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:37.794304 2636 consensus_queue.cc:1035] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:37.794397 2654 consensus_queue.cc:1035] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:09:37.799882 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:09:39.321060 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:39.321226 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:39.322220 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Sending DeleteTablet for 3 replicas of tablet 974fb070617f4121b8dfaf6b6614b8d6
I20260924 00:09:39.322497 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 00:09:39.322527 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260924 00:09:39.322547 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 974fb070617f4121b8dfaf6b6614b8d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34264
I20260924 00:09:39.324059 2710 tablet_replica.cc:330] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: stopping tablet replica
I20260924 00:09:39.324138 2710 raft_consensus.cc:2229] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:39.324187 2710 raft_consensus.cc:2258] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.324483 2710 ts_tablet_manager.cc:1905] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.325218 2712 tablet_replica.cc:330] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: stopping tablet replica
I20260924 00:09:39.325296 2712 raft_consensus.cc:2229] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:39.325361 2712 raft_consensus.cc:2258] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.325474 2711 tablet_replica.cc:330] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: stopping tablet replica
I20260924 00:09:39.325538 2711 raft_consensus.cc:2229] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:39.325593 2712 ts_tablet_manager.cc:1905] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.325639 2711 raft_consensus.cc:2258] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.325878 2711 ts_tablet_manager.cc:1905] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.326169 2710 ts_tablet_manager.cc:1918] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:09:39.326236 2710 log.cc:1196] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/974fb070617f4121b8dfaf6b6614b8d6
I20260924 00:09:39.326476 2710 ts_tablet_manager.cc:1939] T 974fb070617f4121b8dfaf6b6614b8d6 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting consensus metadata
I20260924 00:09:39.326871 2110 catalog_manager.cc:4841] TS 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913): tablet 974fb070617f4121b8dfaf6b6614b8d6 (table test-table [id=023e85e2c2fe40f6b87b133f648c4179]) successfully deleted
I20260924 00:09:39.327222 2712 ts_tablet_manager.cc:1918] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:09:39.327308 2712 log.cc:1196] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/974fb070617f4121b8dfaf6b6614b8d6
I20260924 00:09:39.327329 2711 ts_tablet_manager.cc:1918] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:09:39.327415 2711 log.cc:1196] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/974fb070617f4121b8dfaf6b6614b8d6
I20260924 00:09:39.327539 2712 ts_tablet_manager.cc:1939] T 974fb070617f4121b8dfaf6b6614b8d6 P 365beb42c86d434c95c3142a85000cfb: Deleting consensus metadata
I20260924 00:09:39.327692 2711 ts_tablet_manager.cc:1939] T 974fb070617f4121b8dfaf6b6614b8d6 P af791e73e3aa4f518fe2f611620e8882: Deleting consensus metadata
I20260924 00:09:39.327900 2111 catalog_manager.cc:4841] TS 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475): tablet 974fb070617f4121b8dfaf6b6614b8d6 (table test-table [id=023e85e2c2fe40f6b87b133f648c4179]) successfully deleted
I20260924 00:09:39.327975 2109 catalog_manager.cc:4841] TS af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485): tablet 974fb070617f4121b8dfaf6b6614b8d6 (table test-table [id=023e85e2c2fe40f6b87b133f648c4179]) successfully deleted
Sep 24 00:09:39 dist-test-slave-3q8k krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:09:39 dist-test-slave-3q8k krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:39.356500 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.342027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.0.98.62:33423)
0924 00:09:39.342187 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.342189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.342198 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:39.342240 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.342241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.342251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.342302 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:39.342853 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.343425 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.344193 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.344379 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.344430 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.353618 (+ 9188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:39.353620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:39.353621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:39.353653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:39.355185 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:39.355602 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:39.355604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:39.355604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:39.355658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:39.355848 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:39.355849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:39.355849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:39.355960 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:39.356050 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.356284 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.356385 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260924 00:09:39.358038 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:39.358210 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:39.359687 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd2ed4b29ac464083aa444f29e666be (DEFAULT_TABLE table=test-table [id=ce78408bf9b345f58d5705be045b2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.359723 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd2ed4b29ac464083aa444f29e666be (DEFAULT_TABLE table=test-table [id=ce78408bf9b345f58d5705be045b2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.359853 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd2ed4b29ac464083aa444f29e666be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.359875 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd2ed4b29ac464083aa444f29e666be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.360906 2721 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Bootstrap starting.
I20260924 00:09:39.360963 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd2ed4b29ac464083aa444f29e666be (DEFAULT_TABLE table=test-table [id=ce78408bf9b345f58d5705be045b2226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.361088 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd2ed4b29ac464083aa444f29e666be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.361253 2721 tablet_bootstrap.cc:654] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.361902 2721 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: No bootstrap required, opened a new log
I20260924 00:09:39.361964 2721 ts_tablet_manager.cc:1397] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:39.362097 2721 raft_consensus.cc:348] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.362177 2721 raft_consensus.cc:374] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.362200 2721 raft_consensus.cc:729] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.362248 2721 consensus_queue.cc:260] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.362433 2721 ts_tablet_manager.cc:1428] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:39.362746 2725 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Bootstrap starting.
I20260924 00:09:39.363132 2725 tablet_bootstrap.cc:654] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.363334 2722 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Bootstrap starting.
I20260924 00:09:39.363670 2722 tablet_bootstrap.cc:654] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.364238 2725 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: No bootstrap required, opened a new log
I20260924 00:09:39.364300 2725 ts_tablet_manager.cc:1397] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:39.364449 2725 raft_consensus.cc:348] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.364537 2725 raft_consensus.cc:374] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.364552 2725 raft_consensus.cc:729] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.364634 2725 consensus_queue.cc:260] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.364764 2725 ts_tablet_manager.cc:1428] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:39.364934 2722 tablet_bootstrap.cc:492] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: No bootstrap required, opened a new log
I20260924 00:09:39.364992 2722 ts_tablet_manager.cc:1397] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:39.365177 2722 raft_consensus.cc:348] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.365275 2722 raft_consensus.cc:374] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.365315 2722 raft_consensus.cc:729] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.365379 2722 consensus_queue.cc:260] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.365559 2722 ts_tablet_manager.cc:1428] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:39.409088 2698 raft_consensus.cc:482] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:39.409184 2698 raft_consensus.cc:504] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.409313 2698 leader_election.cc:288] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:39.409533 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd2ed4b29ac464083aa444f29e666be" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb" is_pre_election: true
I20260924 00:09:39.409541 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd2ed4b29ac464083aa444f29e666be" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" is_pre_election: true
I20260924 00:09:39.409638 2561 raft_consensus.cc:2454] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:39.409643 2280 raft_consensus.cc:2454] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 0.
I20260924 00:09:39.409816 2354 leader_election.cc:302] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a387c5b443c47eea5a2b0fe6a6d43e4, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:39.409911 2698 raft_consensus.cc:2781] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:39.409946 2698 raft_consensus.cc:482] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:39.409966 2698 raft_consensus.cc:3037] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.410656 2698 raft_consensus.cc:504] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.410791 2698 leader_election.cc:288] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Requested vote from peers 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:39.410928 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd2ed4b29ac464083aa444f29e666be" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365beb42c86d434c95c3142a85000cfb"
I20260924 00:09:39.411005 2280 raft_consensus.cc:3037] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.410972 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd2ed4b29ac464083aa444f29e666be" candidate_uuid: "af791e73e3aa4f518fe2f611620e8882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
I20260924 00:09:39.411073 2561 raft_consensus.cc:3037] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.411835 2561 raft_consensus.cc:2454] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:39.411988 2280 raft_consensus.cc:2454] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af791e73e3aa4f518fe2f611620e8882 in term 1.
I20260924 00:09:39.412061 2354 leader_election.cc:302] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a387c5b443c47eea5a2b0fe6a6d43e4, af791e73e3aa4f518fe2f611620e8882; no voters:
I20260924 00:09:39.412148 2698 raft_consensus.cc:2781] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:39.412235 2698 raft_consensus.cc:686] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 1 LEADER]: Becoming Leader. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Running, Role: LEADER
I20260924 00:09:39.412289 2698 consensus_queue.cc:237] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.412894 2122 catalog_manager.cc:5495] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 reported cstate change: term changed from 0 to 1, leader changed from <none> to af791e73e3aa4f518fe2f611620e8882 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af791e73e3aa4f518fe2f611620e8882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:39.423810 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.421557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.0.98.2:42485)
0924 00:09:39.421696 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.421697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.421718 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:09:39.421736 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.421737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.421764 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.421831 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:39.421941 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.422428 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.423154 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.423413 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.423465 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.423581 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.423608 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.423645 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.423674 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.423690 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260924 00:09:39.425906 2561 raft_consensus.cc:1261] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:39.425918 2280 raft_consensus.cc:1261] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Refusing update from remote peer af791e73e3aa4f518fe2f611620e8882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:39.426168 2698 consensus_queue.cc:1035] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:39.426376 2675 consensus_queue.cc:1035] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:39.435918 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:39.436017 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:39.436640 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Sending DeleteTablet for 3 replicas of tablet 4dd2ed4b29ac464083aa444f29e666be
I20260924 00:09:39.436872 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd2ed4b29ac464083aa444f29e666be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260924 00:09:39.436872 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd2ed4b29ac464083aa444f29e666be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34264
I20260924 00:09:39.436923 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd2ed4b29ac464083aa444f29e666be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 00:09:39.436962 2712 tablet_replica.cc:330] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: stopping tablet replica
I20260924 00:09:39.437007 2712 raft_consensus.cc:2229] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:39.437026 2711 tablet_replica.cc:330] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: stopping tablet replica
I20260924 00:09:39.437052 2712 raft_consensus.cc:2258] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.437090 2711 raft_consensus.cc:2229] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:39.437152 2711 raft_consensus.cc:2258] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.437120 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:39.437250 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:39.437288 2710 tablet_replica.cc:330] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: stopping tablet replica
I20260924 00:09:39.437345 2710 raft_consensus.cc:2229] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:39.437386 2710 raft_consensus.cc:2258] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:39.437490 2711 ts_tablet_manager.cc:1905] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.437634 2712 ts_tablet_manager.cc:1905] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.438320 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 (DEFAULT_TABLE table=test-table [id=4f0a0c2defb7409aa975c82aac2c5849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.438385 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 (DEFAULT_TABLE table=test-table [id=4f0a0c2defb7409aa975c82aac2c5849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.438436 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abbf8e5abd40e3a2819783da5fea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.438473 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abbf8e5abd40e3a2819783da5fea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.439395 2712 ts_tablet_manager.cc:1918] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:39.439466 2712 log.cc:1196] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/4dd2ed4b29ac464083aa444f29e666be
I20260924 00:09:39.439684 2722 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb: Bootstrap starting.
I20260924 00:09:39.439719 2712 ts_tablet_manager.cc:1939] T 4dd2ed4b29ac464083aa444f29e666be P 365beb42c86d434c95c3142a85000cfb: Deleting consensus metadata
I20260924 00:09:39.439843 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 (DEFAULT_TABLE table=test-table [id=4f0a0c2defb7409aa975c82aac2c5849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:39.440004 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70abbf8e5abd40e3a2819783da5fea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:39.440114 2722 tablet_bootstrap.cc:654] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.440222 2710 ts_tablet_manager.cc:1905] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:39.440455 2721 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Bootstrap starting.
I20260924 00:09:39.440778 2721 tablet_bootstrap.cc:654] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.441138 2111 catalog_manager.cc:4841] TS 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475): tablet 4dd2ed4b29ac464083aa444f29e666be (table test-table [id=ce78408bf9b345f58d5705be045b2226]) successfully deleted
I20260924 00:09:39.441294 2725 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Bootstrap starting.
I20260924 00:09:39.441358 2721 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: No bootstrap required, opened a new log
I20260924 00:09:39.441397 2721 ts_tablet_manager.cc:1397] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:39.441502 2721 raft_consensus.cc:348] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.441592 2721 raft_consensus.cc:374] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.441594 2725 tablet_bootstrap.cc:654] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:39.441617 2721 raft_consensus.cc:729] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a387c5b443c47eea5a2b0fe6a6d43e4, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.441673 2721 consensus_queue.cc:260] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.441771 2710 ts_tablet_manager.cc:1918] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:39.441787 2721 ts_tablet_manager.cc:1428] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:39.441846 2710 log.cc:1196] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/4dd2ed4b29ac464083aa444f29e666be
I20260924 00:09:39.441965 2722 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb: No bootstrap required, opened a new log
I20260924 00:09:39.442011 2722 ts_tablet_manager.cc:1397] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:39.440013 2711 ts_tablet_manager.cc:1918] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:09:39.442083 2711 log.cc:1196] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/4dd2ed4b29ac464083aa444f29e666be
I20260924 00:09:39.442114 2710 ts_tablet_manager.cc:1939] T 4dd2ed4b29ac464083aa444f29e666be P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting consensus metadata
I20260924 00:09:39.442189 2722 raft_consensus.cc:348] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.442270 2722 raft_consensus.cc:374] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.442296 2722 raft_consensus.cc:729] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.442338 2711 ts_tablet_manager.cc:1939] T 4dd2ed4b29ac464083aa444f29e666be P af791e73e3aa4f518fe2f611620e8882: Deleting consensus metadata
I20260924 00:09:39.442339 2722 consensus_queue.cc:260] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.442391 2110 catalog_manager.cc:4841] TS 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913): tablet 4dd2ed4b29ac464083aa444f29e666be (table test-table [id=ce78408bf9b345f58d5705be045b2226]) successfully deleted
I20260924 00:09:39.442444 2722 ts_tablet_manager.cc:1428] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:39.442675 2109 catalog_manager.cc:4841] TS af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485): tablet 4dd2ed4b29ac464083aa444f29e666be (table test-table [id=ce78408bf9b345f58d5705be045b2226]) successfully deleted
I20260924 00:09:39.442822 2725 tablet_bootstrap.cc:492] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: No bootstrap required, opened a new log
I20260924 00:09:39.442873 2725 ts_tablet_manager.cc:1397] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:39.442999 2725 raft_consensus.cc:348] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.443076 2725 raft_consensus.cc:374] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:39.443100 2725 raft_consensus.cc:729] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af791e73e3aa4f518fe2f611620e8882, State: Initialized, Role: FOLLOWER
I20260924 00:09:39.443142 2725 consensus_queue.cc:260] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.443259 2725 ts_tablet_manager.cc:1428] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:39.502936 2728 raft_consensus.cc:482] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:39.503042 2728 raft_consensus.cc:504] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.503378 2728 leader_election.cc:288] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:39.507718 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.503721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59571 (local address 127.0.98.3:44913)
0924 00:09:39.503858 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.503861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.503869 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:39.503900 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.503901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.503913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.503988 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:39.504269 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.504970 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.506623 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.507251 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.507345 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.507358 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.507560 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.507631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
I20260924 00:09:39.507707 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.503785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50445 (local address 127.0.98.2:42485)
0924 00:09:39.503840 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.503842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.503886 (+ 44us) server_negotiation.cc:407] Connection header received
0924 00:09:39.503912 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.503913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.503925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.503984 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:39.504213 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.504985 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.506623 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.507289 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.507353 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.507363 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.507560 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.507625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:09:39.507701 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.503577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44913 (local address 127.0.98.1:59571)
0924 00:09:39.503726 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:39.503738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:39.503795 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:39.504010 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:39.504011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:39.504014 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:39.504123 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.504125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.505092 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.505092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.506479 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.506483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.507359 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.507360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.507445 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.507448 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:39.507513 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:09:39.507570 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":59,"thread_start_us":78,"threads_started":1}
I20260924 00:09:39.507709 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.503577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.98.1:50445)
0924 00:09:39.503785 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:39.503794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:39.503835 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:39.504010 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:39.504011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:39.504013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:39.504123 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.504125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.505073 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.505075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.506479 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.506483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.507360 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.507360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.507443 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.507450 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:09:39.507513 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:09:39.507570 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":145,"thread_start_us":79,"threads_started":1}
I20260924 00:09:39.507970 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abbf8e5abd40e3a2819783da5fea58" candidate_uuid: "365beb42c86d434c95c3142a85000cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" is_pre_election: true
I20260924 00:09:39.508074 2561 raft_consensus.cc:2454] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365beb42c86d434c95c3142a85000cfb in term 0.
I20260924 00:09:39.508204 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abbf8e5abd40e3a2819783da5fea58" candidate_uuid: "365beb42c86d434c95c3142a85000cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af791e73e3aa4f518fe2f611620e8882" is_pre_election: true
I20260924 00:09:39.508297 2421 raft_consensus.cc:2454] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365beb42c86d434c95c3142a85000cfb in term 0.
I20260924 00:09:39.508483 2214 leader_election.cc:302] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365beb42c86d434c95c3142a85000cfb, 6a387c5b443c47eea5a2b0fe6a6d43e4; no voters:
I20260924 00:09:39.508679 2728 raft_consensus.cc:2781] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:39.508720 2728 raft_consensus.cc:482] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:39.508735 2728 raft_consensus.cc:3037] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.509567 2728 raft_consensus.cc:504] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.509727 2728 leader_election.cc:288] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [CANDIDATE]: Term 1 election: Requested vote from peers af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485), 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913)
I20260924 00:09:39.509881 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abbf8e5abd40e3a2819783da5fea58" candidate_uuid: "365beb42c86d434c95c3142a85000cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4"
I20260924 00:09:39.509873 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70abbf8e5abd40e3a2819783da5fea58" candidate_uuid: "365beb42c86d434c95c3142a85000cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af791e73e3aa4f518fe2f611620e8882"
I20260924 00:09:39.509951 2561 raft_consensus.cc:3037] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.509953 2421 raft_consensus.cc:3037] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:39.510542 2561 raft_consensus.cc:2454] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365beb42c86d434c95c3142a85000cfb in term 1.
I20260924 00:09:39.510599 2421 raft_consensus.cc:2454] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365beb42c86d434c95c3142a85000cfb in term 1.
I20260924 00:09:39.510746 2214 leader_election.cc:302] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365beb42c86d434c95c3142a85000cfb, 6a387c5b443c47eea5a2b0fe6a6d43e4; no voters:
I20260924 00:09:39.510855 2728 raft_consensus.cc:2781] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:39.511004 2728 raft_consensus.cc:686] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [term 1 LEADER]: Becoming Leader. State: Replica: 365beb42c86d434c95c3142a85000cfb, State: Running, Role: LEADER
I20260924 00:09:39.511061 2728 consensus_queue.cc:237] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } }
I20260924 00:09:39.511878 2122 catalog_manager.cc:5495] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 365beb42c86d434c95c3142a85000cfb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "365beb42c86d434c95c3142a85000cfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "365beb42c86d434c95c3142a85000cfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:39.521332 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.517776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.98.1:39475)
0924 00:09:39.518328 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.518330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.518337 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:09:39.518368 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.518368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.518386 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.518867 (+ 481us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:39.519184 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.519728 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.520479 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.520693 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.520768 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.520958 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.521062 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.521153 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.521193 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.521213 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":483,"threads_started":1}
I20260924 00:09:39.526348 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.524184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36847 (local address 127.0.98.2:42485)
0924 00:09:39.524265 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.524267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.524331 (+ 64us) server_negotiation.cc:407] Connection header received
0924 00:09:39.524360 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.524361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.524371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.524423 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:39.524522 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.524988 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.525735 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.525918 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.525976 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.526152 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.526177 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.526215 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.526266 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.526285 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:09:39.526366 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.524101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42485 (local address 127.0.98.1:36847)
0924 00:09:39.524263 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:39.524274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:39.524326 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:39.524426 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:39.524427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:39.524427 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:39.524473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.524474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.525060 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.525061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.525654 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.525656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.526003 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.526004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.526058 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.526071 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.526228 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.526230 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:39.526263 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 00:09:39.526300 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 00:09:39.528772 2421 raft_consensus.cc:1261] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Refusing update from remote peer 365beb42c86d434c95c3142a85000cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:39.528839 2561 raft_consensus.cc:1261] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Refusing update from remote peer 365beb42c86d434c95c3142a85000cfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:39.529091 2728 consensus_queue.cc:1035] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a387c5b443c47eea5a2b0fe6a6d43e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:39.529223 2741 consensus_queue.cc:1035] T 70abbf8e5abd40e3a2819783da5fea58 P 365beb42c86d434c95c3142a85000cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "af791e73e3aa4f518fe2f611620e8882" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:39.536409 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.533821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45229 (local address 127.0.98.1:39475)
0924 00:09:39.533873 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:39.533875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:39.533992 (+ 117us) server_negotiation.cc:407] Connection header received
0924 00:09:39.534066 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:39.534067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:39.534078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:39.534133 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:39.534268 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.534725 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.535694 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.535984 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.536034 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.536175 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.536211 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.536257 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:39.536305 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 00:09:39.536325 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:09:39.536443 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:39.533778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39475 (local address 127.0.98.2:45229)
0924 00:09:39.533937 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:39.533948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:39.534007 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:39.534161 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:39.534162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:39.534165 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:09:39.534214 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.534216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.534790 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.534791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.535612 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:39.535615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:39.536038 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:39.536039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:39.536081 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:39.536085 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:09:39.536256 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:09:39.536258 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:39.536304 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 00:09:39.536351 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":114,"threads_started":1}
W20260924 00:09:39.537698 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:09:41.061657 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:41.061759 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:41.062688 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e57af915a5c41efb1b789417cf9470b: Sending DeleteTablet for 3 replicas of tablet 70abbf8e5abd40e3a2819783da5fea58
I20260924 00:09:41.062959 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47670
I20260924 00:09:41.062960 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34264
I20260924 00:09:41.063148 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 70abbf8e5abd40e3a2819783da5fea58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 00:09:41.063230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2196
I20260924 00:09:41.063972 2776 tablet_replica.cc:330] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: stopping tablet replica
I20260924 00:09:41.064038 2776 raft_consensus.cc:2229] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:41.064087 2776 raft_consensus.cc:2258] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:41.064321 2776 ts_tablet_manager.cc:1905] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:41.065903 2776 ts_tablet_manager.cc:1918] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:09:41.065955 2776 log.cc:1196] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/70abbf8e5abd40e3a2819783da5fea58
I20260924 00:09:41.066172 2776 ts_tablet_manager.cc:1939] T 70abbf8e5abd40e3a2819783da5fea58 P 6a387c5b443c47eea5a2b0fe6a6d43e4: Deleting consensus metadata
I20260924 00:09:41.066498 2110 catalog_manager.cc:4841] TS 6a387c5b443c47eea5a2b0fe6a6d43e4 (127.0.98.3:44913): tablet 70abbf8e5abd40e3a2819783da5fea58 (table test-table [id=4f0a0c2defb7409aa975c82aac2c5849]) successfully deleted
I20260924 00:09:41.066638 2774 tablet_replica.cc:330] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: stopping tablet replica
I20260924 00:09:41.066706 2774 raft_consensus.cc:2229] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:41.066752 2774 raft_consensus.cc:2258] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:41.067013 2774 ts_tablet_manager.cc:1905] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:41.068395 2774 ts_tablet_manager.cc:1918] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:09:41.068471 2774 log.cc:1196] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/70abbf8e5abd40e3a2819783da5fea58
I20260924 00:09:41.068703 2774 ts_tablet_manager.cc:1939] T 70abbf8e5abd40e3a2819783da5fea58 P af791e73e3aa4f518fe2f611620e8882: Deleting consensus metadata
I20260924 00:09:41.069067 2109 catalog_manager.cc:4841] TS af791e73e3aa4f518fe2f611620e8882 (127.0.98.2:42485): tablet 70abbf8e5abd40e3a2819783da5fea58 (table test-table [id=4f0a0c2defb7409aa975c82aac2c5849]) successfully deleted
W20260924 00:09:41.071792 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39475 (error 108)
I20260924 00:09:41.072062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2336
I20260924 00:09:41.072878 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:41.072182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39475 (local address 127.0.0.1:47698)
0924 00:09:41.072773 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:41.072829 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":61,"threads_started":1}
W20260924 00:09:41.073029 2111 catalog_manager.cc:4584] TS 365beb42c86d434c95c3142a85000cfb (127.0.98.1:39475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70abbf8e5abd40e3a2819783da5fea58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39475: connect: Connection refused (error 111)
I20260924 00:09:41.078886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2476
I20260924 00:09:41.084371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2092
2026-09-24T00:09:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6152 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:41 dist-test-slave-3q8k krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:41 dist-test-slave-3q8k krb5kdc[2781](info): set up 2 sockets
Sep 24 00:09:41 dist-test-slave-3q8k krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260924 00:09:42.119071 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:42.238307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:42.318357 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:42.318612 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:42.318650 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:42.319865 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:42.319941 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:42.319967 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:42.319998 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:42.320029 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:42.321583 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:42.321874 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:42.322105 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:09:42.327486 2794 server_base.cc:1034] running on GCE node
W20260924 00:09:42.327497 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.327492 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.327669 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:42.327955 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:42.328258 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:42.329420 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790208582329412 us; error 51 us; skew 500 ppm
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:42.332070 2794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:42.332567 2794 webserver.cc:460] Webserver started at http://127.0.98.62:40437/ using document root <none> and password file <none>
I20260924 00:09:42.332813 2794 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:42.332885 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:42.333043 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:42.334177 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "9071b7af6ee7487d82db12efc88f6dc7"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "3ba7ff8b7d1c093868e5e9bd486fb319"
server_key_iv: "786c66598429e56b8dbb917d6e01b089"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.334550 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "9071b7af6ee7487d82db12efc88f6dc7"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "3ba7ff8b7d1c093868e5e9bd486fb319"
server_key_iv: "786c66598429e56b8dbb917d6e01b089"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.336000 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:42.336771 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.336961 2794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:42.337069 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "9071b7af6ee7487d82db12efc88f6dc7"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "3ba7ff8b7d1c093868e5e9bd486fb319"
server_key_iv: "786c66598429e56b8dbb917d6e01b089"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.337145 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:42.344138 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:42.346520 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:42.346683 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:42.351097 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40797
I20260924 00:09:42.351506 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40797 every 8 connection(s)
I20260924 00:09:42.351557 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:42.352295 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:42.353153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2794
I20260924 00:09:42.353261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:42.353437 392 external_mini_cluster.cc:1428] Setting key 118dd5a15736231242cfc39762459933
I20260924 00:09:42.355662 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7: Bootstrap starting.
I20260924 00:09:42.356279 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:42.356576 2862 log.cc:824] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:42.357347 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7: No bootstrap required, opened a new log
I20260924 00:09:42.359889 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260924 00:09:42.360028 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:42.360050 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9071b7af6ee7487d82db12efc88f6dc7, State: Initialized, Role: FOLLOWER
I20260924 00:09:42.360185 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260924 00:09:42.360267 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:42.360292 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:42.360359 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:42.360991 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260924 00:09:42.361099 2862 leader_election.cc:302] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9071b7af6ee7487d82db12efc88f6dc7; no voters:
I20260924 00:09:42.361285 2862 leader_election.cc:288] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:42.361374 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:42.361519 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9071b7af6ee7487d82db12efc88f6dc7, State: Running, Role: LEADER
I20260924 00:09:42.361552 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:42.361702 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260924 00:09:42.362066 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9071b7af6ee7487d82db12efc88f6dc7. Latest consensus state: current_term: 1 leader_uuid: "9071b7af6ee7487d82db12efc88f6dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260924 00:09:42.362077 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9071b7af6ee7487d82db12efc88f6dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9071b7af6ee7487d82db12efc88f6dc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260924 00:09:42.362737 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:42.362943 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:42.362977 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:42.363571 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:42.365271 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.353980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.0.98.62:40797)
0924 00:09:42.354144 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:42.354153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:42.354189 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:09:42.354584 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:42.354591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:42.354636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:42.354750 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:42.355578 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.356554 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.357575 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.357991 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.358076 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.361015 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:42.361019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:42.361020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:42.361032 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:42.363232 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.363940 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.363944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.363944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.364049 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.364274 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.364275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.364275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.364513 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:42.364609 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:42.364908 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 00:09:42.365066 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 00:09:42.365725 2877 catalog_manager.cc:1347] Generated new cluster ID: c6c079a37d374d40a5b2fe6deeb50ad0
I20260924 00:09:42.365768 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:42.381217 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:42.381984 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:42.394340 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9071b7af6ee7487d82db12efc88f6dc7: Generated new TSK 0
I20260924 00:09:42.394624 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:42.421628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:42.504601 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:42.504860 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:42.504926 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:42.506104 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:42.506183 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:42.506229 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:42.507992 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:42.508277 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:42.508522 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:42.513722 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.513742 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.513717 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:42.514029 2888 server_base.cc:1034] running on GCE node
I20260924 00:09:42.514243 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:42.514479 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:42.515645 2888 hybrid_clock.cc:648] HybridClock initialized: now 1790208582515644 us; error 42 us; skew 500 ppm
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:42.518244 2888 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:42.518729 2888 webserver.cc:460] Webserver started at http://127.0.98.1:37745/ using document root <none> and password file <none>
I20260924 00:09:42.518959 2888 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:42.519028 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:42.519182 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:42.520313 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ac8fd29a8c3475bbeae9fb86ac26bd5"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "cc1c0c456d1ad074a2b7480b90b248ca"
server_key_iv: "fa5183e8425640af062a77ccd8ed66b5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.520694 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ac8fd29a8c3475bbeae9fb86ac26bd5"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "cc1c0c456d1ad074a2b7480b90b248ca"
server_key_iv: "fa5183e8425640af062a77ccd8ed66b5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.522056 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:42.522790 2903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.522987 2888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:42.523094 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "9ac8fd29a8c3475bbeae9fb86ac26bd5"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "cc1c0c456d1ad074a2b7480b90b248ca"
server_key_iv: "fa5183e8425640af062a77ccd8ed66b5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.523170 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:42.532147 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:42.534590 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:42.534765 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:42.535025 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:42.535507 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:42.535566 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.535600 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:42.535629 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.541762 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41817
I20260924 00:09:42.541785 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41817 every 8 connection(s)
I20260924 00:09:42.542171 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:42.546782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 2888
I20260924 00:09:42.546864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:42.547058 392 external_mini_cluster.cc:1428] Setting key e636266f4730fa5e889d6221ba9862e0
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:42.553561 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.542950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34755 (local address 127.0.98.62:40797)
0924 00:09:42.543032 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:42.543035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:42.543321 (+ 286us) server_negotiation.cc:407] Connection header received
0924 00:09:42.544332 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:42.544333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:42.544343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:42.544408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:42.545056 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.545588 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.546605 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.546812 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.546922 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.549834 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:42.549836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:42.549836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:42.549844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:42.551917 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.552428 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.552430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.552430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.552503 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.552744 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.552745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.552746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.552910 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:42.552983 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:42.553321 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 00:09:42.553420 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:09:42.553596 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.542921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.1:34755)
0924 00:09:42.543105 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:42.543144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:42.544209 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:42.544466 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:42.544469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:42.544737 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:42.544958 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.544961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.545679 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.545680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.546506 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.546508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.546942 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.546943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.547102 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.547758 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:42.547770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:42.549725 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:42.552012 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.552019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.552021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.552334 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.552575 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.552575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.552576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.552680 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.553019 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:42.553025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:42.553169 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:09:42.553271 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 00:09:42.553409 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260924 00:09:42.554320 3017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260924 00:09:42.554420 3017 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:42.554639 3017 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260924 00:09:42.555271 2826 ts_manager.cc:194] Registered new tserver with Master: 9ac8fd29a8c3475bbeae9fb86ac26bd5 (127.0.98.1:41817)
I20260924 00:09:42.556277 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34755
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:42.574769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:42.657075 3022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:42.657317 3022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:42.657342 3022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:42.658457 3022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:42.658494 3022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:42.658520 3022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:42.660054 3022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:42.660259 3022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:42.660445 3022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:42.665831 3030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.665884 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.665850 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:42.666087 3022 server_base.cc:1034] running on GCE node
I20260924 00:09:42.666352 3022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:42.666612 3022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:42.667788 3022 hybrid_clock.cc:648] HybridClock initialized: now 1790208582667784 us; error 40 us; skew 500 ppm
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:42.670424 3022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:42.670826 3022 webserver.cc:460] Webserver started at http://127.0.98.2:44115/ using document root <none> and password file <none>
I20260924 00:09:42.671006 3022 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:42.671046 3022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:42.671135 3022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:42.672226 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "6355bd6064314b62aa0072a8b15b8890"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "7529b1c096da1f8d5d0ce3cc3018f55c"
server_key_iv: "c2de51492ca034251584a18b029e34c4"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.672533 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6355bd6064314b62aa0072a8b15b8890"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "7529b1c096da1f8d5d0ce3cc3018f55c"
server_key_iv: "c2de51492ca034251584a18b029e34c4"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.673806 3022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:09:42.674510 3037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.674681 3022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:42.674765 3022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "6355bd6064314b62aa0072a8b15b8890"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "7529b1c096da1f8d5d0ce3cc3018f55c"
server_key_iv: "c2de51492ca034251584a18b029e34c4"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.674822 3022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:42.698633 3022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:42.700837 3022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:42.700977 3022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:42.701184 3022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:42.701524 3022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:42.701561 3022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.701586 3022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:42.701597 3022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.707671 3022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40683
I20260924 00:09:42.707754 3150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40683 every 8 connection(s)
I20260924 00:09:42.708103 3022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:42.710140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3022
I20260924 00:09:42.710270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:42.710474 392 external_mini_cluster.cc:1428] Setting key 5f039beabcf035a77726c9e61a32df76
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:42.720055 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.708961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33199 (local address 127.0.98.62:40797)
0924 00:09:42.709031 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:42.709033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:42.709253 (+ 220us) server_negotiation.cc:407] Connection header received
0924 00:09:42.710348 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:42.710349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:42.710370 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:42.710457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:42.711458 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.712223 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.713700 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.713922 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.713979 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.716752 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:42.716753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:42.716754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:42.716760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:42.718189 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.718837 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.718838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.718839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.718891 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.719267 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.719267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.719268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.719396 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:42.719453 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:42.719922 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 00:09:42.719987 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:09:42.720377 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.708945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.2:33199)
0924 00:09:42.709108 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:42.709153 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:42.710236 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:42.710568 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:42.710573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:42.710989 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:42.711362 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.711367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.712323 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.712324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.713612 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.713615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.714004 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.714005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.714157 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.714703 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:42.714715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:42.716630 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:42.718416 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.718424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.718427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.718754 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.718985 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.718986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.718986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.719141 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.719494 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:42.719502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:42.719754 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 00:09:42.719901 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 00:09:42.720087 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 00:09:42.720911 3151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260924 00:09:42.721005 3151 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:42.721267 3151 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260924 00:09:42.721786 2826 ts_manager.cc:194] Registered new tserver with Master: 6355bd6064314b62aa0072a8b15b8890 (127.0.98.2:40683)
I20260924 00:09:42.722393 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:42.738339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:42.819298 3156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:42.819552 3156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:42.819589 3156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:42.820730 3156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:42.820804 3156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:42.820850 3156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:42.822358 3156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:42.822633 3156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:42.822883 3156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:42.827898 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.827929 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:42.827929 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:42.828094 3156 server_base.cc:1034] running on GCE node
I20260924 00:09:42.828310 3156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:42.828533 3156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:42.829674 3156 hybrid_clock.cc:648] HybridClock initialized: now 1790208582829653 us; error 42 us; skew 500 ppm
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:42.832329 3156 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:42.832763 3156 webserver.cc:460] Webserver started at http://127.0.98.3:37103/ using document root <none> and password file <none>
I20260924 00:09:42.833043 3156 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:42.833112 3156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:42.833276 3156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:42.834342 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ee95dc3e2d64202b4013c3aaf9e96ca"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "d6740781ad0b1beb8485a775e4e81c72"
server_key_iv: "83f7cae1f7d97dc1a35426652b9229a5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.834708 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee95dc3e2d64202b4013c3aaf9e96ca"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "d6740781ad0b1beb8485a775e4e81c72"
server_key_iv: "83f7cae1f7d97dc1a35426652b9229a5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.836194 3156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:42.836938 3171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.837134 3156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:42.837246 3156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "6ee95dc3e2d64202b4013c3aaf9e96ca"
format_stamp: "Formatted at 2026-09-24 00:09:42 on dist-test-slave-3q8k"
server_key: "d6740781ad0b1beb8485a775e4e81c72"
server_key_iv: "83f7cae1f7d97dc1a35426652b9229a5"
server_key_version: "encryptionkey@0"
I20260924 00:09:42.837344 3156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:42.859735 3156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:42.862226 3156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:42.862392 3156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:42.862623 3156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:42.862996 3156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:42.863049 3156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.863080 3156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:42.863097 3156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:42.868916 3156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42745
I20260924 00:09:42.868957 3284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42745 every 8 connection(s)
I20260924 00:09:42.869288 3156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:42.873615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3156
I20260924 00:09:42.873718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:42.873881 392 external_mini_cluster.cc:1428] Setting key fc5e2dab872131c1aeaf8d5fcec23658
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:42.878835 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.869990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35045 (local address 127.0.98.62:40797)
0924 00:09:42.870058 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:42.870059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:42.870310 (+ 251us) server_negotiation.cc:407] Connection header received
0924 00:09:42.871317 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:42.871318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:42.871328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:42.871379 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:42.871973 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.872483 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.873506 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.873686 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.873770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.876226 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:42.876228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:42.876228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:42.876234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:42.877506 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.877957 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.877958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.877958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.878009 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:42.878239 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:42.878240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:42.878240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:42.878390 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:42.878449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:42.878701 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 00:09:42.878765 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:42.879076 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:42.869971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.3:35045)
0924 00:09:42.870148 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:42.870191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:42.871174 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:42.871438 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:42.871442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:42.871699 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:42.871900 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.871904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.872571 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.872572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.873398 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:42.873401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:42.873868 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:42.873869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:42.874010 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:42.874619 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:42.874630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:42.876145 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:42.877643 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.877650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.877652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.877890 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.878071 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:42.878072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:42.878072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:42.878172 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:42.878452 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:42.878456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:42.878575 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 00:09:42.878661 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:09:42.878836 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260924 00:09:42.879560 3285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260924 00:09:42.879669 3285 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:42.879866 3285 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260924 00:09:42.880244 2826 ts_manager.cc:194] Registered new tserver with Master: 6ee95dc3e2d64202b4013c3aaf9e96ca (127.0.98.3:42745)
I20260924 00:09:42.880573 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35045
I20260924 00:09:42.885712 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:09:42.896448 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:42.895306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.0.98.62:40797)
0924 00:09:42.895371 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:42.895373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:42.895546 (+ 173us) server_negotiation.cc:407] Connection header received
0924 00:09:42.895592 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:42.895593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:42.895617 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:42.895668 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:42.896372 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33604: BlockingRecv error: recv got EOF from 127.0.0.1:33604 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:09:42.897661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2888
I20260924 00:09:42.908823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3022
I20260924 00:09:42.914145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3156
I20260924 00:09:42.918388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 2794
2026-09-24T00:09:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1832 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[3297](info): set up 2 sockets
Sep 24 00:09:42 dist-test-slave-3q8k krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260924 00:09:43.959293 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:44.079365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:44.160425 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:44.160691 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:44.160733 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:44.161921 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:44.161998 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:44.162021 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:44.162042 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:44.162074 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:44.163642 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:44.163934 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:44.164180 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:44.169056 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:44.169067 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:44.169214 3310 server_base.cc:1034] running on GCE node
W20260924 00:09:44.169067 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:44.169564 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:44.169864 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:44.171031 3310 hybrid_clock.cc:648] HybridClock initialized: now 1790208584171019 us; error 45 us; skew 500 ppm
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:44.173650 3310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:44.174082 3310 webserver.cc:460] Webserver started at http://127.0.98.62:34783/ using document root <none> and password file <none>
I20260924 00:09:44.174331 3310 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:44.174408 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:44.174556 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:44.175752 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "5320c0e8838643f78cd34908d0e97139"
format_stamp: "Formatted at 2026-09-24 00:09:44 on dist-test-slave-3q8k"
server_key: "425e0adce3835a3760c06f70a305f41b"
server_key_iv: "87dd4a4823e695ac9a3d6751e6ca5424"
server_key_version: "encryptionkey@0"
I20260924 00:09:44.176107 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "5320c0e8838643f78cd34908d0e97139"
format_stamp: "Formatted at 2026-09-24 00:09:44 on dist-test-slave-3q8k"
server_key: "425e0adce3835a3760c06f70a305f41b"
server_key_iv: "87dd4a4823e695ac9a3d6751e6ca5424"
server_key_version: "encryptionkey@0"
I20260924 00:09:44.177492 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:44.178236 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:44.178406 3310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:44.178514 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "5320c0e8838643f78cd34908d0e97139"
format_stamp: "Formatted at 2026-09-24 00:09:44 on dist-test-slave-3q8k"
server_key: "425e0adce3835a3760c06f70a305f41b"
server_key_iv: "87dd4a4823e695ac9a3d6751e6ca5424"
server_key_version: "encryptionkey@0"
I20260924 00:09:44.178589 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:44.199795 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:44.202275 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:44.202458 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:44.206593 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45693
I20260924 00:09:44.206636 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45693 every 8 connection(s)
I20260924 00:09:44.207031 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:44.207918 3378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:44.210213 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139: Bootstrap starting.
I20260924 00:09:44.210835 3378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:44.211131 3378 log.cc:824] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:44.211858 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139: No bootstrap required, opened a new log
I20260924 00:09:44.213682 3378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } }
I20260924 00:09:44.213836 3378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:44.213860 3378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5320c0e8838643f78cd34908d0e97139, State: Initialized, Role: FOLLOWER
I20260924 00:09:44.214040 3378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } }
I20260924 00:09:44.214123 3378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:44.214184 3378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:44.214233 3378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:44.214226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3310
I20260924 00:09:44.214318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:44.214483 392 external_mini_cluster.cc:1428] Setting key 687420f6c9a9701d4aea455a892fde31
I20260924 00:09:44.215332 3378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } }
I20260924 00:09:44.215453 3378 leader_election.cc:302] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5320c0e8838643f78cd34908d0e97139; no voters:
I20260924 00:09:44.215619 3378 leader_election.cc:288] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:44.215651 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:44.215788 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [term 1 LEADER]: Becoming Leader. State: Replica: 5320c0e8838643f78cd34908d0e97139, State: Running, Role: LEADER
I20260924 00:09:44.215876 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } }
I20260924 00:09:44.215901 3378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:44.216200 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5320c0e8838643f78cd34908d0e97139. Latest consensus state: current_term: 1 leader_uuid: "5320c0e8838643f78cd34908d0e97139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } }
I20260924 00:09:44.216213 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5320c0e8838643f78cd34908d0e97139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5320c0e8838643f78cd34908d0e97139" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } }
I20260924 00:09:44.216292 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:44.216300 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:44.216480 3388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:44.217043 3388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:44.218988 3388 catalog_manager.cc:1347] Generated new cluster ID: 2c51bd3b6c7e4b48b526636aad17a017
I20260924 00:09:44.219043 3388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:44.226040 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:44.215015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.0.98.62:45693)
0924 00:09:44.215208 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:44.215247 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:44.215280 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:09:44.215391 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:44.215396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:44.215458 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:44.215627 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:44.216956 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:44.218432 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:44.219498 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:44.219949 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:44.220084 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:44.222513 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:44.222517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:44.222518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:44.222532 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:44.224468 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:44.224864 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:44.224867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:44.224867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:44.224967 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:44.225175 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:44.225176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:44.225177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:44.225424 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:44.225607 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:44.225675 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:09:44.225831 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260924 00:09:44.226893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:09:44.251325 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:44.247725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.98.62:45693)
0924 00:09:44.247774 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:44.247777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:44.247919 (+ 142us) server_negotiation.cc:407] Connection header received
0924 00:09:44.248003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:44.248004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:44.248017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:44.248090 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:44.248480 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:44.248979 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:44.249734 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:44.249945 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:44.250033 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:44.250781 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:44.250782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:44.250782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:44.251001 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:44.251018 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:44.251024 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:44.251103 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:44.251157 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:09:44.251183 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:44.258548 3388 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:44.259089 3388 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:44.268316 3388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5320c0e8838643f78cd34908d0e97139: Generated new TSK 0
I20260924 00:09:44.268541 3388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:09:44.272909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3310
2026-09-24T00:09:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1352 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3411](info): set up 2 sockets
Sep 24 00:09:44 dist-test-slave-3q8k krb5kdc[3411](info): commencing operation
krb5kdc: starting...
W20260924 00:09:45.311036 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:45.429622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:45.509646 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:45.509863 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:45.509887 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:45.511065 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:45.511112 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:45.511127 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:45.511171 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:45.511200 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:45.512741 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:45.512938 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:45.513108 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:45.518116 3430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:45.518098 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:45.518119 3432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:45.518400 3424 server_base.cc:1034] running on GCE node
I20260924 00:09:45.518649 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:45.518910 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:45.520051 3424 hybrid_clock.cc:648] HybridClock initialized: now 1790208585520036 us; error 31 us; skew 500 ppm
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:45.530831 3424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:45.531447 3424 webserver.cc:460] Webserver started at http://127.0.98.62:43679/ using document root <none> and password file <none>
I20260924 00:09:45.531711 3424 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:45.531782 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:45.531940 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:45.533043 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "9e7a04e4fad04f598cc83aac6a33e9b6"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "cb7005a955e2b9fe663915d81c099249"
server_key_iv: "b236a7574cc824dbc572c33e9655857a"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.533433 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e7a04e4fad04f598cc83aac6a33e9b6"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "cb7005a955e2b9fe663915d81c099249"
server_key_iv: "b236a7574cc824dbc572c33e9655857a"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.534785 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:45.535645 3439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.535844 3424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:09:45.535961 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "9e7a04e4fad04f598cc83aac6a33e9b6"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "cb7005a955e2b9fe663915d81c099249"
server_key_iv: "b236a7574cc824dbc572c33e9655857a"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.536041 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:45.551947 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:45.554005 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:45.554179 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:45.558492 3424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44829
I20260924 00:09:45.558576 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44829 every 8 connection(s)
I20260924 00:09:45.558955 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:45.560029 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:45.562758 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6: Bootstrap starting.
I20260924 00:09:45.563524 3492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:45.563829 3492 log.cc:824] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:45.564391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3424
I20260924 00:09:45.564495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:45.564685 392 external_mini_cluster.cc:1428] Setting key e15a2f837fc893d44c133ff23623b863
I20260924 00:09:45.564705 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6: No bootstrap required, opened a new log
I20260924 00:09:45.566656 3492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } }
I20260924 00:09:45.566829 3492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:45.566880 3492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7a04e4fad04f598cc83aac6a33e9b6, State: Initialized, Role: FOLLOWER
I20260924 00:09:45.567068 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } }
I20260924 00:09:45.567170 3492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:45.567238 3492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:45.567312 3492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:45.567896 3492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } }
I20260924 00:09:45.568015 3492 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7a04e4fad04f598cc83aac6a33e9b6; no voters:
I20260924 00:09:45.568229 3492 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:45.568251 3497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:45.568365 3497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [term 1 LEADER]: Becoming Leader. State: Replica: 9e7a04e4fad04f598cc83aac6a33e9b6, State: Running, Role: LEADER
I20260924 00:09:45.568447 3497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } }
I20260924 00:09:45.568509 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:45.568778 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } } }
I20260924 00:09:45.568866 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:45.569020 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7a04e4fad04f598cc83aac6a33e9b6. Latest consensus state: current_term: 1 leader_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7a04e4fad04f598cc83aac6a33e9b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44829 } } }
I20260924 00:09:45.569073 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:45.569166 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:45.569659 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:45.571141 3514 mvcc.cc:205] Tried to move back new op lower bound from 7332694366497198080 to 7332694366488731648. Current Snapshot: MvccSnapshot[applied={T|T < 7332694366497198080}]
I20260924 00:09:45.571563 3501 catalog_manager.cc:1347] Generated new cluster ID: cfb369ef7f7a461ebe0fbf8cf40826dc
I20260924 00:09:45.571612 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:45.576267 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:45.576831 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:45.565219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.62:44829)
0924 00:09:45.565415 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:45.565424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:45.565450 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:09:45.565944 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:45.565952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:45.566001 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:45.566161 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:45.566818 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.568313 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.569429 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.570120 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.570231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:45.573040 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:45.573044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:45.573045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:45.573057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:45.575120 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.575550 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.575554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.575554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.575666 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.575889 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.575890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.575890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.576230 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:45.576425 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:45.576496 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:09:45.576628 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260924 00:09:45.576928 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:45.583624 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7a04e4fad04f598cc83aac6a33e9b6: Generated new TSK 0
I20260924 00:09:45.583808 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:45.610246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:45.692158 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:45.692368 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:45.692392 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:45.693576 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:45.693624 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:45.693650 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:45.695210 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:45.695479 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:45.695729 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:45.700863 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:45.701020 3518 server_base.cc:1034] running on GCE node
W20260924 00:09:45.700793 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:45.700793 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:45.701347 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:45.701615 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:45.702782 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790208585702768 us; error 44 us; skew 500 ppm
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:45.705394 3518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:45.705843 3518 webserver.cc:460] Webserver started at http://127.0.98.1:42205/ using document root <none> and password file <none>
I20260924 00:09:45.706092 3518 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:45.706161 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:45.706314 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:45.707510 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c64d61a3af14860a5cbe6b8d26d659e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "3759612c24364877bb2ef8998805034a"
server_key_iv: "fcbc0959c219e922c926442a27436277"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.707885 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c64d61a3af14860a5cbe6b8d26d659e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "3759612c24364877bb2ef8998805034a"
server_key_iv: "fcbc0959c219e922c926442a27436277"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.709201 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:45.709985 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.710279 3518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:45.710383 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "7c64d61a3af14860a5cbe6b8d26d659e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "3759612c24364877bb2ef8998805034a"
server_key_iv: "fcbc0959c219e922c926442a27436277"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.710458 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:45.728278 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:45.730217 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:45.730391 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:45.730647 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:45.730970 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:45.731026 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.731068 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:45.731112 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.736946 3518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45301
I20260924 00:09:45.736984 3646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45301 every 8 connection(s)
I20260924 00:09:45.737322 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:45.745147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3518
I20260924 00:09:45.745231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:45.745465 392 external_mini_cluster.cc:1428] Setting key 1d734b060e1c625d9104d2b3a22f2960
I20260924 00:09:45.747051 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:45.738095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51373 (local address 127.0.98.62:44829)
0924 00:09:45.738155 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:45.738157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:45.738375 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:09:45.738977 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:45.738978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:45.738991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:45.739042 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:45.740131 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.740638 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.741655 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.741872 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.741965 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:45.744100 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:45.744102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:45.744102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:45.744110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:45.745469 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.745912 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.745913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.745913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.745977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.746292 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.746293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.746293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.746461 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:45.746530 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:45.746871 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 00:09:45.746986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:09:45.747437 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:45.738065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44829 (local address 127.0.98.1:51373)
0924 00:09:45.738228 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:45.738288 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:45.738838 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:45.739110 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:45.739114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:45.739773 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:45.740032 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:45.740038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.740735 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.740736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:45.741556 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:45.741559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.741974 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.741975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:45.742113 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:45.742606 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:45.742614 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:45.743995 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:45.745578 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:45.745585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:45.745588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:45.745836 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:45.746064 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:45.746065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:45.746065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:45.746221 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:45.746543 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:45.746549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:45.746722 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 00:09:45.746845 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 00:09:45.747033 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 00:09:45.747995 3647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44829
I20260924 00:09:45.748103 3647 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:45.748332 3647 heartbeater.cc:511] Master 127.0.98.62:44829 requested a full tablet report, sending...
I20260924 00:09:45.748880 3456 ts_manager.cc:194] Registered new tserver with Master: 7c64d61a3af14860a5cbe6b8d26d659e (127.0.98.1:45301)
I20260924 00:09:45.749830 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51373
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:45.770058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:45.851699 3652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:45.851898 3652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:45.851922 3652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:45.852998 3652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:45.853036 3652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:45.853062 3652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:45.854497 3652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:45.854676 3652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:45.854851 3652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:45.860013 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:45.859992 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:45.859972 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:45.860215 3652 server_base.cc:1034] running on GCE node
I20260924 00:09:45.860502 3652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:45.860755 3652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:45.861928 3652 hybrid_clock.cc:648] HybridClock initialized: now 1790208585861916 us; error 31 us; skew 500 ppm
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:45.864621 3652 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:45.865021 3652 webserver.cc:460] Webserver started at http://127.0.98.2:43407/ using document root <none> and password file <none>
I20260924 00:09:45.865218 3652 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:45.865259 3652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:45.865356 3652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:45.866384 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "f4e17334d25146d397f3ade040851f4e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "f0dac68987c4b06e4b894ae76248fe9a"
server_key_iv: "c5517e956f26027ed7440817955ceabf"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.866689 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f4e17334d25146d397f3ade040851f4e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "f0dac68987c4b06e4b894ae76248fe9a"
server_key_iv: "c5517e956f26027ed7440817955ceabf"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.868046 3652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:45.868889 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.869066 3652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:45.869174 3652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "f4e17334d25146d397f3ade040851f4e"
format_stamp: "Formatted at 2026-09-24 00:09:45 on dist-test-slave-3q8k"
server_key: "f0dac68987c4b06e4b894ae76248fe9a"
server_key_iv: "c5517e956f26027ed7440817955ceabf"
server_key_version: "encryptionkey@0"
I20260924 00:09:45.869234 3652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:45.878957 3652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:45.881295 3652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:45.881405 3652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:45.881583 3652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:45.881822 3652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:45.881852 3652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.881873 3652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:45.881886 3652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:45.887683 3652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38843
I20260924 00:09:45.887763 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38843 every 8 connection(s)
I20260924 00:09:45.888069 3652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:09:45 dist-test-slave-3q8k krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:45.894822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3652
I20260924 00:09:45.894950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:45.895275 392 external_mini_cluster.cc:1428] Setting key daf0eca3adee9a4461a360cd4862d4b0
I20260924 00:09:45.898200 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:45.888819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41403 (local address 127.0.98.62:44829)
0924 00:09:45.888869 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:45.888871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:45.889074 (+ 203us) server_negotiation.cc:407] Connection header received
0924 00:09:45.889573 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:45.889574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:45.889583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:45.889642 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:45.890536 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.891071 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.892182 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.892368 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.892439 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:45.894666 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:45.894667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:45.894668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:45.894675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:45.896545 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.896948 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.896950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.896950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.897016 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:45.897418 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:45.897420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:45.897420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:45.897565 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:45.897651 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:45.897988 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 00:09:45.898094 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:09:45.898545 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:45.888782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44829 (local address 127.0.98.2:41403)
0924 00:09:45.888943 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:45.888985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:45.889486 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:45.889696 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:45.889699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:45.890268 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:45.890457 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:45.890461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.891159 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.891160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:45.892095 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:45.892098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:45.892482 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:45.892483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:45.892602 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:45.893117 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:45.893125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:45.894554 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:45.896634 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:45.896639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:45.896641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:45.896870 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:45.897123 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:45.897124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:45.897125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:45.897308 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:45.897665 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:45.897673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:45.897834 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 00:09:45.897952 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 00:09:45.898147 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260924 00:09:45.899107 3781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44829
I20260924 00:09:45.899258 3781 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:45.899485 3781 heartbeater.cc:511] Master 127.0.98.62:44829 requested a full tablet report, sending...
I20260924 00:09:45.899926 3456 ts_manager.cc:194] Registered new tserver with Master: f4e17334d25146d397f3ade040851f4e (127.0.98.2:38843)
I20260924 00:09:45.900341 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41403
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:45.922842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:46.002614 3786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:46.002840 3786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:46.002863 3786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:46.004069 3786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:46.004115 3786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:46.004174 3786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:46.005584 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:46.005831 3786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:46.006059 3786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:46.011551 3794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:46.011582 3792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:46.011562 3791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:46.011775 3786 server_base.cc:1034] running on GCE node
I20260924 00:09:46.011981 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:46.012249 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:46.013469 3786 hybrid_clock.cc:648] HybridClock initialized: now 1790208586013446 us; error 32 us; skew 500 ppm
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:46.016141 3786 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:46.016546 3786 webserver.cc:460] Webserver started at http://127.0.98.3:44951/ using document root <none> and password file <none>
I20260924 00:09:46.016784 3786 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:46.016830 3786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:46.017005 3786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:46.018172 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "a0910051d6a248c798cdb1b26ba6f0cf"
format_stamp: "Formatted at 2026-09-24 00:09:46 on dist-test-slave-3q8k"
server_key: "ba560e2099a938c2746d95c797695339"
server_key_iv: "d6b027f329200000776d185f8dccc07e"
server_key_version: "encryptionkey@0"
I20260924 00:09:46.018541 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a0910051d6a248c798cdb1b26ba6f0cf"
format_stamp: "Formatted at 2026-09-24 00:09:46 on dist-test-slave-3q8k"
server_key: "ba560e2099a938c2746d95c797695339"
server_key_iv: "d6b027f329200000776d185f8dccc07e"
server_key_version: "encryptionkey@0"
I20260924 00:09:46.019990 3786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:46.020824 3801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:46.021031 3786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:09:46.021124 3786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "a0910051d6a248c798cdb1b26ba6f0cf"
format_stamp: "Formatted at 2026-09-24 00:09:46 on dist-test-slave-3q8k"
server_key: "ba560e2099a938c2746d95c797695339"
server_key_iv: "d6b027f329200000776d185f8dccc07e"
server_key_version: "encryptionkey@0"
I20260924 00:09:46.021205 3786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:46.032078 3786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:46.034085 3786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:46.034228 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:46.034483 3786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:46.034798 3786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:46.034853 3786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:46.034904 3786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:46.034933 3786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:46.041536 3786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35847
I20260924 00:09:46.041599 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35847 every 8 connection(s)
I20260924 00:09:46.041936 3786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:46.047844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3786
I20260924 00:09:46.047966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:46.048149 392 external_mini_cluster.cc:1428] Setting key 907c240ab38312e85e47bfedbd437913
I20260924 00:09:46.051481 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:46.042705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36501 (local address 127.0.98.62:44829)
0924 00:09:46.042783 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:46.042786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:46.042960 (+ 174us) server_negotiation.cc:407] Connection header received
0924 00:09:46.043539 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:46.043540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:46.043550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:46.043601 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:46.044527 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.045031 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.046030 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.046229 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.046288 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:46.048698 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:46.048699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:46.048699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:46.048708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:46.050157 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:46.050559 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:46.050561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:46.050561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:46.050623 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:46.050866 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:46.050867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:46.050867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:46.050982 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:46.051038 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:46.051336 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 00:09:46.051416 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:09:46.051672 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:46.042666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44829 (local address 127.0.98.3:36501)
0924 00:09:46.042837 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:46.042872 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:46.043445 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:46.043662 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:46.043666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:46.044241 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:46.044431 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:46.044435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.045112 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.045112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:46.045952 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:46.045954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.046413 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.046414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:46.046522 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:46.046982 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:46.046990 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:46.048575 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:46.050232 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:46.050238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:46.050240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:46.050469 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:46.050694 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:46.050695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:46.050695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:46.050799 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:46.051039 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:46.051042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:46.051155 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 00:09:46.051312 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 00:09:46.051458 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260924 00:09:46.052126 3915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44829
I20260924 00:09:46.052232 3915 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:46.052424 3915 heartbeater.cc:511] Master 127.0.98.62:44829 requested a full tablet report, sending...
I20260924 00:09:46.052791 3456 ts_manager.cc:194] Registered new tserver with Master: a0910051d6a248c798cdb1b26ba6f0cf (127.0.98.3:35847)
I20260924 00:09:46.053122 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36501
I20260924 00:09:46.059998 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208586, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208586, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:46.082237 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:46.075520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.0.98.62:44829)
0924 00:09:46.075579 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:46.075581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:46.075679 (+ 98us) server_negotiation.cc:407] Connection header received
0924 00:09:46.075736 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:46.075737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:46.075746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:46.075802 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:46.076330 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.076875 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.077575 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:46.077752 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:46.077804 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:46.079984 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:46.079985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:46.079985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:46.079991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:46.081213 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:46.081576 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:46.081576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:46.081577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:46.081623 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:46.081797 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:46.081797 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:46.081798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:46.081940 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:46.082015 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:46.082110 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:09:46.082169 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 00:09:46.082535 3456 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33748
I20260924 00:09:46.082957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3518
I20260924 00:09:46.087962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3652
I20260924 00:09:46.092386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3786
I20260924 00:09:46.096575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3424
2026-09-24T00:09:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1825 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3927](info): set up 2 sockets
Sep 24 00:09:46 dist-test-slave-3q8k krb5kdc[3927](info): commencing operation
krb5kdc: starting...
W20260924 00:09:47.135280 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:47.253753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:47.334041 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:47.334306 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:47.334344 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:47.335537 3940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:47.335608 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:47.335631 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:47.335652 3940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:47.335683 3940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:47.337131 3940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:47.337419 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:47.337642 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:47.342512 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.342527 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.342576 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:47.342887 3940 server_base.cc:1034] running on GCE node
I20260924 00:09:47.343175 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:47.343493 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:47.344643 3940 hybrid_clock.cc:648] HybridClock initialized: now 1790208587344629 us; error 27 us; skew 500 ppm
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:47.347055 3940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:47.347563 3940 webserver.cc:460] Webserver started at http://127.0.98.62:39743/ using document root <none> and password file <none>
I20260924 00:09:47.347788 3940 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:47.347855 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:47.348026 3940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:47.349152 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "7798457f7c5349eab286b0f4a72a643c"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "2c25f50780f966851dbdd6333fa15327"
server_key_iv: "e6bf2b94e44bb4e3e3cdf0cda5a29b76"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.349529 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "7798457f7c5349eab286b0f4a72a643c"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "2c25f50780f966851dbdd6333fa15327"
server_key_iv: "e6bf2b94e44bb4e3e3cdf0cda5a29b76"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.350847 3940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:09:47.351641 3955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.351807 3940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.351924 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7798457f7c5349eab286b0f4a72a643c"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "2c25f50780f966851dbdd6333fa15327"
server_key_iv: "e6bf2b94e44bb4e3e3cdf0cda5a29b76"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.352005 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:47.363729 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:47.366117 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:47.366369 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:47.370405 3940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35277
I20260924 00:09:47.370441 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35277 every 8 connection(s)
I20260924 00:09:47.370795 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:47.371842 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:47.374085 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c: Bootstrap starting.
I20260924 00:09:47.374684 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:47.374981 4008 log.cc:824] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:47.375741 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c: No bootstrap required, opened a new log
I20260924 00:09:47.377302 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } }
I20260924 00:09:47.377439 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:47.377461 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798457f7c5349eab286b0f4a72a643c, State: Initialized, Role: FOLLOWER
I20260924 00:09:47.377640 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } }
I20260924 00:09:47.377702 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:47.377745 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:47.377795 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:47.378335 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } }
I20260924 00:09:47.378415 4008 leader_election.cc:302] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7798457f7c5349eab286b0f4a72a643c; no voters:
I20260924 00:09:47.378508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 3940
I20260924 00:09:47.378604 4008 leader_election.cc:288] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:47.378614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:47.378788 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:47.378818 392 external_mini_cluster.cc:1428] Setting key 060fdf2daad34caf3797fc19158b790d
I20260924 00:09:47.378885 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:47.379154 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [term 1 LEADER]: Becoming Leader. State: Replica: 7798457f7c5349eab286b0f4a72a643c, State: Running, Role: LEADER
I20260924 00:09:47.379268 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } }
I20260924 00:09:47.379771 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7798457f7c5349eab286b0f4a72a643c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } } }
I20260924 00:09:47.379859 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:47.380023 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7798457f7c5349eab286b0f4a72a643c. Latest consensus state: current_term: 1 leader_uuid: "7798457f7c5349eab286b0f4a72a643c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7798457f7c5349eab286b0f4a72a643c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35277 } } }
I20260924 00:09:47.380086 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:47.380309 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:47.380875 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:47.382650 4022 catalog_manager.cc:1347] Generated new cluster ID: a63015d9d85a44038cafbd6a3aa8d513
I20260924 00:09:47.382709 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:47.390585 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:47.379491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.62:35277)
0924 00:09:47.379767 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:47.379776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:47.379808 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:09:47.379930 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:47.379938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:47.380007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:47.381480 (+ 1473us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:47.381597 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.382920 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.383767 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.384204 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.384333 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:47.386654 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:47.386658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:47.386658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:47.386671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:47.388942 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.389332 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.389343 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.389343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.389435 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.389650 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.389651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.389652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.389898 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:47.390080 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:47.390177 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 00:09:47.390381 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260924 00:09:47.398556 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:47.399199 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:47.407378 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7798457f7c5349eab286b0f4a72a643c: Generated new TSK 0
I20260924 00:09:47.407601 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:47.439518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:47.521137 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:47.521382 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:47.521435 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:47.522661 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:47.522732 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:47.522778 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:47.524391 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:47.524681 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:47.524930 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:47.530169 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.530155 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.530157 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:47.530825 4034 server_base.cc:1034] running on GCE node
I20260924 00:09:47.531054 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:47.531389 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:47.532580 4034 hybrid_clock.cc:648] HybridClock initialized: now 1790208587532571 us; error 38 us; skew 500 ppm
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:47.535437 4034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:47.535938 4034 webserver.cc:460] Webserver started at http://127.0.98.1:44545/ using document root <none> and password file <none>
I20260924 00:09:47.536201 4034 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:47.536278 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:47.536437 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:47.537595 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "f267d6baf2724601bc66a5de62711847"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "ddb7308e234fb7caefce6f683fdd4b14"
server_key_iv: "1bcf9e7f4bd8f64b1ecff9e6ddb59863"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.538003 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f267d6baf2724601bc66a5de62711847"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "ddb7308e234fb7caefce6f683fdd4b14"
server_key_iv: "1bcf9e7f4bd8f64b1ecff9e6ddb59863"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.539487 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:47.540247 4049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.540410 4034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:47.540511 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "f267d6baf2724601bc66a5de62711847"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "ddb7308e234fb7caefce6f683fdd4b14"
server_key_iv: "1bcf9e7f4bd8f64b1ecff9e6ddb59863"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.540589 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:47.552697 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:47.554811 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:47.554987 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:47.555359 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:47.555737 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:47.555787 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.555833 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:47.555868 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.562206 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45881
I20260924 00:09:47.562245 4162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45881 every 8 connection(s)
I20260924 00:09:47.562606 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:47.564579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4034
I20260924 00:09:47.564780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:47.565035 392 external_mini_cluster.cc:1428] Setting key f79d1aa409659de0c5e4454215f7613e
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:09:47.577374 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:47.563759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35995 (local address 127.0.98.62:35277)
0924 00:09:47.563830 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:47.563832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:47.563863 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:09:47.565315 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:47.565317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:47.565336 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:47.565398 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:47.567973 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.568799 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.570701 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.571022 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.571073 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:47.574227 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:47.574229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:47.574230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:47.574239 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:47.576008 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.576405 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.576406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.576406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.576461 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.576698 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.576698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.576698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.576812 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:47.576868 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:47.577129 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 00:09:47.577304 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:09:47.577394 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:47.563533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35277 (local address 127.0.98.1:35995)
0924 00:09:47.563719 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:47.563772 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:47.565171 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:47.565545 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:47.565551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:47.567447 (+ 1896us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:47.567712 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:47.567718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.568926 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.568927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:47.570572 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:47.570576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.571191 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.571192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:47.571416 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:47.572148 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:47.572160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:47.574106 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:47.576104 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:47.576111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:47.576113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:47.576335 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:47.576526 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:47.576526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:47.576527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:47.576632 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:47.576871 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:47.576875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:47.577010 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 00:09:47.577096 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:09:47.577219 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260924 00:09:47.577955 4163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35277
I20260924 00:09:47.578059 4163 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:47.578260 4163 heartbeater.cc:511] Master 127.0.98.62:35277 requested a full tablet report, sending...
I20260924 00:09:47.578836 3972 ts_manager.cc:194] Registered new tserver with Master: f267d6baf2724601bc66a5de62711847 (127.0.98.1:45881)
I20260924 00:09:47.579672 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 00:09:47.716286 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.150s user 0.002s sys 0.001s
I20260924 00:09:47.716617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:47.798420 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:47.798629 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:47.798653 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:47.799793 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:47.799834 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:47.799860 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:47.801307 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:47.801494 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:47.801664 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:09:47.806617 4168 server_base.cc:1034] running on GCE node
W20260924 00:09:47.806548 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.806542 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.806548 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:47.807051 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:47.807349 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:47.808514 4168 hybrid_clock.cc:648] HybridClock initialized: now 1790208587808510 us; error 37 us; skew 500 ppm
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:47.811036 4168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:47.811527 4168 webserver.cc:460] Webserver started at http://127.0.98.2:34141/ using document root <none> and password file <none>
I20260924 00:09:47.811764 4168 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:47.811838 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:47.811992 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:47.813016 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ab84236da8a4cac959bd5ab98120936"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "0abe88c4344b6236f2d6d7ecd426efdf"
server_key_iv: "4c9f787ba88dc0bfce9f98ea498322df"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.813371 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ab84236da8a4cac959bd5ab98120936"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "0abe88c4344b6236f2d6d7ecd426efdf"
server_key_iv: "4c9f787ba88dc0bfce9f98ea498322df"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.814670 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:47.815460 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.815681 4168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:47.815778 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "9ab84236da8a4cac959bd5ab98120936"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "0abe88c4344b6236f2d6d7ecd426efdf"
server_key_iv: "4c9f787ba88dc0bfce9f98ea498322df"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.815876 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:47.846663 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:47.848667 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:47.848838 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:47.849077 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:47.849397 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:47.849454 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.849496 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:47.849524 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.855680 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33587
I20260924 00:09:47.855736 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33587 every 8 connection(s)
I20260924 00:09:47.856079 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:47.861720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4168
I20260924 00:09:47.861840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:47.862059 392 external_mini_cluster.cc:1428] Setting key 2094a2ee1e61481cd8fcfdc6fe0cc5f5
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:47.866078 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:47.856810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37373 (local address 127.0.98.62:35277)
0924 00:09:47.856867 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:47.856871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:47.857071 (+ 200us) server_negotiation.cc:407] Connection header received
0924 00:09:47.858118 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:47.858119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:47.858129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:47.858178 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:47.858770 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.859355 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.860308 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.860525 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.860580 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:47.862979 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:47.862980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:47.862980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:47.862986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:47.864661 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.865059 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.865060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.865061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.865129 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:47.865383 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:47.865384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:47.865384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:47.865520 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:47.865603 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:47.865877 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 00:09:47.865986 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:09:47.866318 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:47.856800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35277 (local address 127.0.98.2:37373)
0924 00:09:47.856963 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:47.857000 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:47.858025 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:47.858240 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:47.858243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:47.858505 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:47.858701 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:47.858704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.859438 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.859439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:47.860228 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:47.860230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:47.860599 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:47.860600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:47.860697 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:47.861250 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:47.861258 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:47.862889 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:47.864744 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:47.864750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:47.864752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:47.864981 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:47.865204 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:47.865205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:47.865205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:47.865303 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:47.865591 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:47.865597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:47.865741 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:09:47.865857 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 00:09:47.866020 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260924 00:09:47.866824 4297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35277
I20260924 00:09:47.866971 4297 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:47.867173 4297 heartbeater.cc:511] Master 127.0.98.62:35277 requested a full tablet report, sending...
I20260924 00:09:47.867630 3972 ts_manager.cc:194] Registered new tserver with Master: 9ab84236da8a4cac959bd5ab98120936 (127.0.98.2:33587)
I20260924 00:09:47.868008 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37373
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:47.888298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:47.968307 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:47.968535 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:47.968560 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:47.969758 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:47.969805 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:47.969831 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:47.971304 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:47.971553 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:47.971791 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:09:47.976811 4302 server_base.cc:1034] running on GCE node
W20260924 00:09:47.976758 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.976760 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:47.976789 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:47.977213 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:47.977422 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:47.978564 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790208587978551 us; error 31 us; skew 500 ppm
Sep 24 00:09:47 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:47.981401 4302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:47.981859 4302 webserver.cc:460] Webserver started at http://127.0.98.3:34523/ using document root <none> and password file <none>
I20260924 00:09:47.982087 4302 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:47.982157 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:47.982333 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:47.983515 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ae53d9050bd4d86a57375f2559b60b0"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "237880389b6e9fa804df75d768f046e7"
server_key_iv: "860821b8415ceb3a67641c4fa51f75ff"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.983882 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ae53d9050bd4d86a57375f2559b60b0"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "237880389b6e9fa804df75d768f046e7"
server_key_iv: "860821b8415ceb3a67641c4fa51f75ff"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.985189 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:47.985929 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.986114 4302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:47.986198 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "1ae53d9050bd4d86a57375f2559b60b0"
format_stamp: "Formatted at 2026-09-24 00:09:47 on dist-test-slave-3q8k"
server_key: "237880389b6e9fa804df75d768f046e7"
server_key_iv: "860821b8415ceb3a67641c4fa51f75ff"
server_key_version: "encryptionkey@0"
I20260924 00:09:47.986254 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:47.995522 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:47.997483 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:47.997602 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:47.997805 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:47.998059 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:47.998090 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:47.998112 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:47.998126 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:48.003930 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37605
I20260924 00:09:48.004010 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37605 every 8 connection(s)
I20260924 00:09:48.004338 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:48.012931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4302
I20260924 00:09:48.013041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:48.013208 392 external_mini_cluster.cc:1428] Setting key 0952aa12b144b5822ef55ffd42da6ccd
I20260924 00:09:48.014063 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:48.005260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52829 (local address 127.0.98.62:35277)
0924 00:09:48.005325 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:48.005327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:48.005363 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:09:48.006391 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:48.006391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:48.006403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:48.006470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:48.007104 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.007658 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.008637 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.008814 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.008886 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:48.011209 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:48.011211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:48.011211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:48.011268 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:48.012531 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.012936 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.012937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.012937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.013002 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.013345 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.013346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.013347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.013471 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:48.013555 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:48.013882 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 00:09:48.013969 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:09:48.014278 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:48.005079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35277 (local address 127.0.98.3:52829)
0924 00:09:48.005239 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:48.005283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:48.006206 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:48.006549 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:48.006552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:48.006817 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:48.007013 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:48.007017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.007745 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.007746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:48.008561 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:48.008563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.008928 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.008929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:48.009045 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:48.009716 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:48.009724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:48.011122 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:48.012604 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:48.012610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:48.012612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:48.012854 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:48.013124 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:48.013125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:48.013125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:48.013259 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:48.013546 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:48.013552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:48.013730 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 00:09:48.013857 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:09:48.014026 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260924 00:09:48.014680 4431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35277
I20260924 00:09:48.014782 4431 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:48.014966 4431 heartbeater.cc:511] Master 127.0.98.62:35277 requested a full tablet report, sending...
I20260924 00:09:48.015437 3972 ts_manager.cc:194] Registered new tserver with Master: 1ae53d9050bd4d86a57375f2559b60b0 (127.0.98.3:37605)
I20260924 00:09:48.015836 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52829
I20260924 00:09:48.024849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:48.047734 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:48.040011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.0.98.1:45881)
0924 00:09:48.040232 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:48.040239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:48.040252 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:09:48.040288 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:48.040290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:48.040304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:48.040371 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:48.040917 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.041636 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.042334 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.042665 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.042744 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:48.045088 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:48.045095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:48.045095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:48.045102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:48.046550 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.046907 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.046910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.046910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.046975 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.047138 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.047139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.047139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.047358 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:48.047502 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:48.047590 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:09:48.047653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:48.055881 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:48.049055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.98.62:35277)
0924 00:09:48.049119 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:48.049122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:48.049158 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:09:48.049235 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:48.049236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:48.049252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:48.049309 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:48.049807 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.050287 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.050988 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:48.051172 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:48.051290 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:48.053462 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:48.053464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:48.053464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:48.053470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:48.054802 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.055118 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.055119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.055119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.055207 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:48.055403 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:48.055403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:48.055403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:48.055543 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:48.055616 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:48.055703 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 00:09:48.055761 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 00:09:48.056154 3972 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46932
I20260924 00:09:48.056533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4034
I20260924 00:09:48.061710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4168
I20260924 00:09:48.066335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4302
I20260924 00:09:48.070580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 3940
2026-09-24T00:09:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1973 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[4443](info): set up 2 sockets
Sep 24 00:09:48 dist-test-slave-3q8k krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260924 00:09:49.103116 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:49.222308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:49.301699 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:49.301951 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:49.301995 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:49.303138 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:49.303208 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:49.303263 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:49.303297 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:49.303320 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:49.304822 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:49.305097 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:49.305322 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:49.310094 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:49.310185 4456 server_base.cc:1034] running on GCE node
W20260924 00:09:49.310122 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.310124 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:49.310560 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:49.310847 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:49.312011 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790208589312006 us; error 37 us; skew 500 ppm
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:49.314482 4456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:49.314811 4456 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:49.314880 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:49.315024 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:49.316130 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "b1f410d37eae415287c2220e03c0e163"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "7c67ad2a02775bd9dccb0b2cdf26adba"
server_key_iv: "5fb869ae31b3c7a2ba128440c56a9e34"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.316517 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1f410d37eae415287c2220e03c0e163"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "7c67ad2a02775bd9dccb0b2cdf26adba"
server_key_iv: "5fb869ae31b3c7a2ba128440c56a9e34"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.317768 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:49.318429 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.318595 4456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:49.318702 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "b1f410d37eae415287c2220e03c0e163"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "7c67ad2a02775bd9dccb0b2cdf26adba"
server_key_iv: "5fb869ae31b3c7a2ba128440c56a9e34"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.318773 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:49.327853 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:49.330163 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:49.330330 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:49.334151 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45945
I20260924 00:09:49.334172 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45945 every 8 connection(s)
I20260924 00:09:49.334471 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:49.335446 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:49.336930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4456
I20260924 00:09:49.337039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:49.337226 392 external_mini_cluster.cc:1428] Setting key 564d8700285d71f3f6e12106f50c8790
I20260924 00:09:49.338032 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: Bootstrap starting.
I20260924 00:09:49.338733 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:49.339017 4523 log.cc:824] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:49.339797 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: No bootstrap required, opened a new log
I20260924 00:09:49.341935 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } }
I20260924 00:09:49.342096 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:49.342129 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f410d37eae415287c2220e03c0e163, State: Initialized, Role: FOLLOWER
I20260924 00:09:49.342280 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } }
I20260924 00:09:49.342378 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:49.342446 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:49.342494 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:49.343005 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } }
I20260924 00:09:49.343086 4523 leader_election.cc:302] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f410d37eae415287c2220e03c0e163; no voters:
I20260924 00:09:49.343297 4523 leader_election.cc:288] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:49.343333 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:49.343432 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [term 1 LEADER]: Becoming Leader. State: Replica: b1f410d37eae415287c2220e03c0e163, State: Running, Role: LEADER
I20260924 00:09:49.343509 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } }
I20260924 00:09:49.343564 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:49.343804 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f410d37eae415287c2220e03c0e163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } } }
I20260924 00:09:49.343880 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:49.344002 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f410d37eae415287c2220e03c0e163. Latest consensus state: current_term: 1 leader_uuid: "b1f410d37eae415287c2220e03c0e163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f410d37eae415287c2220e03c0e163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45945 } } }
I20260924 00:09:49.344074 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:49.344206 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:49.344928 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:49.346550 4532 catalog_manager.cc:1347] Generated new cluster ID: fc85f6f42da7447db6041a272d9bdc5c
I20260924 00:09:49.346603 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:49.348692 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.337740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.0.98.62:45945)
0924 00:09:49.337928 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.337937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.337968 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:09:49.338508 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.338516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.338573 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.338737 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:49.339487 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.340823 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.341563 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.342177 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.342294 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.345115 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:49.345119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:49.345120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:49.345132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:49.347130 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.347570 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.347573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.347574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.347668 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.347876 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.347877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.347878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.348116 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:49.348276 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.348382 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.348516 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260924 00:09:49.357491 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:49.358107 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:49.364549 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: Generated new TSK 0
I20260924 00:09:49.364784 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:49.396226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:49.513914 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:49.514194 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:49.514230 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:49.516021 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:49.516078 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:49.516119 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:49.518410 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:49.518685 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:49.518990 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:49.524520 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.524509 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.524510 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:49.524919 4549 server_base.cc:1034] running on GCE node
I20260924 00:09:49.525204 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:49.525491 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:49.526638 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790208589526599 us; error 59 us; skew 500 ppm
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:49.529670 4549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:49.530088 4549 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:49.530154 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:49.530306 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:49.531728 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "9719b7b6d18b4ebe843fa62b5b7acab1"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "05cc3faaf8f1f2fffc53a73ce2b8f48e"
server_key_iv: "2be8d6296429502c78d0b240e0d45d47"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.532158 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9719b7b6d18b4ebe843fa62b5b7acab1"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "05cc3faaf8f1f2fffc53a73ce2b8f48e"
server_key_iv: "2be8d6296429502c78d0b240e0d45d47"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.533581 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:09:49.534464 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.534685 4549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:49.534790 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "9719b7b6d18b4ebe843fa62b5b7acab1"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "05cc3faaf8f1f2fffc53a73ce2b8f48e"
server_key_iv: "2be8d6296429502c78d0b240e0d45d47"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.534874 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:49.549337 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:49.551947 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:49.552111 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:49.552359 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:49.552702 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:49.552757 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.552799 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:49.552829 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.558746 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44989
I20260924 00:09:49.558791 4676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44989 every 8 connection(s)
I20260924 00:09:49.559108 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:49.562085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4549
I20260924 00:09:49.562207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:49.562435 392 external_mini_cluster.cc:1428] Setting key 2fe61580d2dbd8d5d6798d16c892dea4
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:49.569873 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.559888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34905 (local address 127.0.98.62:45945)
0924 00:09:49.559965 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.559967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.560186 (+ 219us) server_negotiation.cc:407] Connection header received
0924 00:09:49.560703 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.560704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.560713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.560767 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:49.561737 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.562334 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.563804 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.564006 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.564064 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.566492 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:49.566494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:49.566494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:49.566504 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:49.568324 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.568772 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.568773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.568774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.568838 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.569092 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.569094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.569094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.569249 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:49.569376 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.569661 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.569753 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:09:49.569931 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.559875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45945 (local address 127.0.98.1:34905)
0924 00:09:49.560060 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.560096 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.560616 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.560839 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.560842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.561098 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:49.561654 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.561660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.562430 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.562431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.563721 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.563724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.564089 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.564090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.564245 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.564875 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:49.564884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:49.566339 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:49.568430 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.568437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.568440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.568681 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.568918 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.568919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.568919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.569023 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.569385 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:49.569390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:49.569518 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.569608 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.569731 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260924 00:09:49.570751 4677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45945
I20260924 00:09:49.570853 4677 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:49.571020 4677 heartbeater.cc:511] Master 127.0.98.62:45945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:09:49.571630 4487 ts_manager.cc:194] Registered new tserver with Master: 9719b7b6d18b4ebe843fa62b5b7acab1 (127.0.98.1:44989)
I20260924 00:09:49.572232 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:49.590378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:49.674932 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:49.675156 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:49.675185 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:49.676502 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:49.676560 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:49.676590 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:49.678236 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:49.678509 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:49.678761 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:49.684376 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.684355 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.684360 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:49.685823 4682 server_base.cc:1034] running on GCE node
I20260924 00:09:49.686071 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:49.686340 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:49.687505 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790208589687489 us; error 28 us; skew 500 ppm
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:49.690908 4682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:49.691406 4682 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:49.691479 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:49.691623 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:49.692804 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "0d5513eae711af9631c68b7b8f222860"
server_key_iv: "70410fb249675c68893379ffdf39376a"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.693288 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "0d5513eae711af9631c68b7b8f222860"
server_key_iv: "70410fb249675c68893379ffdf39376a"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.694785 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:49.695801 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.696000 4682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:09:49.696111 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "0d5513eae711af9631c68b7b8f222860"
server_key_iv: "70410fb249675c68893379ffdf39376a"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.696203 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:49.734364 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:49.736639 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:49.736804 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:49.737038 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:49.737358 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:49.737411 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.737454 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:49.737480 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.743582 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33015
I20260924 00:09:49.743615 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33015 every 8 connection(s)
I20260924 00:09:49.744072 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:49.745879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4682
I20260924 00:09:49.745972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:49.746181 392 external_mini_cluster.cc:1428] Setting key 277f39c0cd3b85bc1beca151a508024a
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:09:49.754264 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.744850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34301 (local address 127.0.98.62:45945)
0924 00:09:49.744920 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.744922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.745112 (+ 190us) server_negotiation.cc:407] Connection header received
0924 00:09:49.745593 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.745594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.745604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.745661 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:49.746832 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.747432 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.748462 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.748664 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.748751 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.751406 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:49.751408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:49.751408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:49.751416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:49.752851 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.753281 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.753283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.753283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.753342 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.753594 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.753595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.753596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.753712 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:49.753789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.754076 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.754153 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:49.754475 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.744855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45945 (local address 127.0.98.2:34301)
0924 00:09:49.745003 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.745040 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.745498 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.745741 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.745745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.746116 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:49.746733 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.746738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.747521 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.747522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.748352 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.748354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.748761 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.748762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.748903 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.749506 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:49.749514 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:49.751170 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:49.752950 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.752958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.752960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.753185 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.753408 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.753408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.753409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.753524 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.753779 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:49.753786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:49.753934 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.754057 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.754221 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260924 00:09:49.754940 4810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45945
I20260924 00:09:49.755096 4810 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:49.755350 4810 heartbeater.cc:511] Master 127.0.98.62:45945 requested a full tablet report, sending...
I20260924 00:09:49.755779 4487 ts_manager.cc:194] Registered new tserver with Master: 01fa13ddfe8e4a12b5fdf601e667bc5f (127.0.98.2:33015)
I20260924 00:09:49.756172 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:49.775259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:49.858063 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:49.858273 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:49.858296 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:49.859460 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:49.859537 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:49.859584 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:49.861037 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:49.861230 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:49.861413 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:49.866643 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.866608 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:49.866658 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:49.866948 4815 server_base.cc:1034] running on GCE node
I20260924 00:09:49.867158 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:49.867449 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:49.868604 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790208589868592 us; error 30 us; skew 500 ppm
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:49.871294 4815 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:49.871608 4815 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:49.871654 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:49.871745 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:49.872785 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2b127b9791a4aae943135c6bdb391ac"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "2e0ba3d658c62e99d3c5b8a249284a65"
server_key_iv: "e4d76f2777dd3c50972eea84a3968560"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.873088 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2b127b9791a4aae943135c6bdb391ac"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "2e0ba3d658c62e99d3c5b8a249284a65"
server_key_iv: "e4d76f2777dd3c50972eea84a3968560"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.874320 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:49.875053 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.875296 4815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:49.875430 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "f2b127b9791a4aae943135c6bdb391ac"
format_stamp: "Formatted at 2026-09-24 00:09:49 on dist-test-slave-3q8k"
server_key: "2e0ba3d658c62e99d3c5b8a249284a65"
server_key_iv: "e4d76f2777dd3c50972eea84a3968560"
server_key_version: "encryptionkey@0"
I20260924 00:09:49.875515 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:49.885412 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:49.887391 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:49.887555 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:49.887785 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:49.888118 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:49.888176 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.888218 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:49.888245 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:49.894536 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40613
I20260924 00:09:49.894603 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40613 every 8 connection(s)
I20260924 00:09:49.894972 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:49.900200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4815
I20260924 00:09:49.900310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:49.900486 392 external_mini_cluster.cc:1428] Setting key 042189fc72ec04b3f9ef92886302604f
Sep 24 00:09:49 dist-test-slave-3q8k krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:49.909574 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.896228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34783 (local address 127.0.98.62:45945)
0924 00:09:49.896346 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.896349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.896611 (+ 262us) server_negotiation.cc:407] Connection header received
0924 00:09:49.897563 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.897564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.897578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.897655 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:49.899315 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.899826 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.901890 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.902200 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.902268 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.905253 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:49.905255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:49.905255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:49.905261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:49.907448 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.908019 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.908020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.908020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.908086 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.908409 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.908410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.908411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.908604 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:49.908916 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.909335 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.909445 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:49.909789 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.896179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45945 (local address 127.0.98.3:34783)
0924 00:09:49.896455 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.896523 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.897422 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.897738 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.897743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.898142 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:49.899182 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.899188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.899940 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.899941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.901534 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.901536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.902296 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.902297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.902467 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.903230 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:49.903241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:49.905151 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:49.907543 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.907551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.907554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.907915 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.908169 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:49.908170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:49.908170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:49.908332 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:49.908930 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:49.908939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:49.909146 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.909304 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.909511 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260924 00:09:49.910542 4943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45945
I20260924 00:09:49.910646 4943 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:49.910818 4943 heartbeater.cc:511] Master 127.0.98.62:45945 requested a full tablet report, sending...
I20260924 00:09:49.911268 4487 ts_manager.cc:194] Registered new tserver with Master: f2b127b9791a4aae943135c6bdb391ac (127.0.98.3:40613)
I20260924 00:09:49.911789 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34783
I20260924 00:09:49.912709 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:09:49.922609 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.914624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.98.62:45945)
0924 00:09:49.914663 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.914666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.914673 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:09:49.914692 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.914693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.914702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.914752 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:49.915544 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.916312 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.917458 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.917636 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.917690 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.919402 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:49.919404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:49.919404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:49.919411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:49.921278 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.921643 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.921644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.921645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.921711 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:49.921969 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:49.921970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:49.921970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:49.922183 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:49.922284 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.922430 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.922504 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:09:49.924402 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:49.925989 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:49.933183 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.928674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.1:44989)
0924 00:09:49.928864 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.928870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.929355 (+ 485us) server_negotiation.cc:407] Connection header received
0924 00:09:49.929412 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.929415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.929429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.929503 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:49.929713 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930430 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.931934 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932679 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932752 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.932816 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.933045 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.933091 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260924 00:09:49.933174 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.928550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40613 (local address 127.0.0.1:35382)
0924 00:09:49.929164 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.929172 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.929207 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.929505 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.929509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.929511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:49.929577 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.929579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930989 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.930990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.932020 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.932023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932875 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.932932 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.932935 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.932978 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.933041 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":5,"thread_start_us":104,"threads_started":1}
I20260924 00:09:49.933188 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.928550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.0.1:44496)
0924 00:09:49.928926 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.928968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.929049 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.929559 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.929560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.929561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:49.929590 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.929591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930952 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.930953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.932142 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.932144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932826 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.932914 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.932927 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.933065 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.933103 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260924 00:09:49.933379 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.928552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44989 (local address 127.0.0.1:33776)
0924 00:09:49.929290 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:49.929298 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:49.929340 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:49.929592 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:49.929593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:49.929595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:49.929652 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.929653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930539 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.930540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.931823 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:49.931826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932852 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:49.932929 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.932936 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:09:49.933025 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:09:49.933056 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260924 00:09:49.933696 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.929017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.0.98.3:40613)
0924 00:09:49.929224 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.929230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.929258 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:09:49.929296 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.929298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.929313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.929389 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:49.929665 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930836 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932123 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932787 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.933148 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.933209 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.933236 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.933583 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260924 00:09:49.933700 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:49.928830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.98.2:33015)
0924 00:09:49.928991 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:49.928995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:49.929024 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:09:49.929159 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:49.929161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:49.929178 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:49.929473 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:49.929664 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.930837 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932226 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:49.932714 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:49.932822 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:49.932881 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:49.933577 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 00:09:49.933624 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260924 00:09:49.934188 4611 tablet_service.cc:1467] Processing CreateTablet for tablet d8560ab60c7641e0ade83974e592bb10 (DEFAULT_TABLE table=test-table [id=7a209b895fa3451abb2d050bdb9c4625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:49.934338 4744 tablet_service.cc:1467] Processing CreateTablet for tablet d8560ab60c7641e0ade83974e592bb10 (DEFAULT_TABLE table=test-table [id=7a209b895fa3451abb2d050bdb9c4625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:49.934320 4877 tablet_service.cc:1467] Processing CreateTablet for tablet d8560ab60c7641e0ade83974e592bb10 (DEFAULT_TABLE table=test-table [id=7a209b895fa3451abb2d050bdb9c4625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:49.934566 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8560ab60c7641e0ade83974e592bb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:49.934625 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8560ab60c7641e0ade83974e592bb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:49.934666 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8560ab60c7641e0ade83974e592bb10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:49.937261 4960 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Bootstrap starting.
I20260924 00:09:49.937929 4962 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Bootstrap starting.
I20260924 00:09:49.938226 4960 tablet_bootstrap.cc:654] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:49.938715 4962 tablet_bootstrap.cc:654] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:49.938855 4960 log.cc:824] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:49.939385 4962 log.cc:824] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:49.939415 4961 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Bootstrap starting.
I20260924 00:09:49.939709 4960 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: No bootstrap required, opened a new log
I20260924 00:09:49.939786 4960 ts_tablet_manager.cc:1397] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:09:49.940034 4961 tablet_bootstrap.cc:654] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:49.940202 4962 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: No bootstrap required, opened a new log
I20260924 00:09:49.940271 4962 ts_tablet_manager.cc:1397] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:49.940630 4961 log.cc:824] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:49.941512 4960 raft_consensus.cc:348] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.941673 4960 raft_consensus.cc:374] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:49.941720 4960 raft_consensus.cc:729] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b127b9791a4aae943135c6bdb391ac, State: Initialized, Role: FOLLOWER
I20260924 00:09:49.941835 4960 consensus_queue.cc:260] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.941942 4962 raft_consensus.cc:348] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.942133 4960 ts_tablet_manager.cc:1428] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:09:49.942152 4962 raft_consensus.cc:374] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:49.942200 4962 raft_consensus.cc:729] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9719b7b6d18b4ebe843fa62b5b7acab1, State: Initialized, Role: FOLLOWER
I20260924 00:09:49.942233 4943 heartbeater.cc:503] Master 127.0.98.62:45945 was elected leader, sending a full tablet report...
I20260924 00:09:49.942305 4962 consensus_queue.cc:260] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.942567 4962 ts_tablet_manager.cc:1428] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:09:49.942576 4677 heartbeater.cc:503] Master 127.0.98.62:45945 was elected leader, sending a full tablet report...
I20260924 00:09:49.942721 4961 tablet_bootstrap.cc:492] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: No bootstrap required, opened a new log
I20260924 00:09:49.942771 4961 ts_tablet_manager.cc:1397] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:09:49.944782 4961 raft_consensus.cc:348] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.944907 4961 raft_consensus.cc:374] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:49.944968 4961 raft_consensus.cc:729] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fa13ddfe8e4a12b5fdf601e667bc5f, State: Initialized, Role: FOLLOWER
I20260924 00:09:49.945083 4961 consensus_queue.cc:260] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:49.945390 4961 ts_tablet_manager.cc:1428] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:09:49.945389 4810 heartbeater.cc:503] Master 127.0.98.62:45945 was elected leader, sending a full tablet report...
W20260924 00:09:49.994603 4811 tablet.cc:2369] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:50.059824 4678 tablet.cc:2369] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:09:50.145860 4944 tablet.cc:2369] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:50.150336 4967 raft_consensus.cc:482] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:50.150465 4967 raft_consensus.cc:504] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:50.150826 4967 leader_election.cc:288] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01fa13ddfe8e4a12b5fdf601e667bc5f (127.0.98.2:33015), f2b127b9791a4aae943135c6bdb391ac (127.0.98.3:40613)
I20260924 00:09:50.153813 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.151076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33691 (local address 127.0.98.3:40613)
0924 00:09:50.151125 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.151128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.151248 (+ 120us) server_negotiation.cc:407] Connection header received
0924 00:09:50.151326 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.151327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.151340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.151416 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:50.151569 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.152045 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153053 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.153429 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153531 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:50.153542 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.153697 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.153730 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:50.153838 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.151101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44731 (local address 127.0.98.2:33015)
0924 00:09:50.151156 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.151158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.151263 (+ 105us) server_negotiation.cc:407] Connection header received
0924 00:09:50.151377 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.151378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.151390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.151473 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:50.151635 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.152095 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153086 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.153456 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153554 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:50.153564 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.153735 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.153769 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:09:50.153862 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.151018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.98.1:44731)
0924 00:09:50.151160 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.151170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.151291 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.151476 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.151477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.151479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:50.151550 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:50.151552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.152188 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.152189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:50.152991 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:50.152993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.153557 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:50.153639 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:50.153644 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.153717 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.153757 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":50,"thread_start_us":68,"threads_started":1}
I20260924 00:09:50.153864 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.151017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40613 (local address 127.0.98.1:33691)
0924 00:09:50.151147 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.151158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.151269 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.151433 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.151434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.151436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:50.151490 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:50.151492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.152138 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.152139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:50.152953 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:50.152955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.153531 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.153532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:50.153595 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:50.153599 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.153688 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.153729 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260924 00:09:50.154227 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8560ab60c7641e0ade83974e592bb10" candidate_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" is_pre_election: true
I20260924 00:09:50.154354 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8560ab60c7641e0ade83974e592bb10" candidate_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b127b9791a4aae943135c6bdb391ac" is_pre_election: true
I20260924 00:09:50.154412 4764 raft_consensus.cc:2454] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719b7b6d18b4ebe843fa62b5b7acab1 in term 0.
I20260924 00:09:50.154477 4897 raft_consensus.cc:2454] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9719b7b6d18b4ebe843fa62b5b7acab1 in term 0.
I20260924 00:09:50.154627 4567 leader_election.cc:302] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fa13ddfe8e4a12b5fdf601e667bc5f, 9719b7b6d18b4ebe843fa62b5b7acab1; no voters:
I20260924 00:09:50.154778 4967 raft_consensus.cc:2781] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:50.154837 4967 raft_consensus.cc:482] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:50.154862 4967 raft_consensus.cc:3037] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.155577 4967 raft_consensus.cc:504] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:50.155763 4967 leader_election.cc:288] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [CANDIDATE]: Term 1 election: Requested vote from peers 01fa13ddfe8e4a12b5fdf601e667bc5f (127.0.98.2:33015), f2b127b9791a4aae943135c6bdb391ac (127.0.98.3:40613)
I20260924 00:09:50.155889 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8560ab60c7641e0ade83974e592bb10" candidate_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f"
I20260924 00:09:50.155993 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8560ab60c7641e0ade83974e592bb10" candidate_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b127b9791a4aae943135c6bdb391ac"
I20260924 00:09:50.156003 4764 raft_consensus.cc:3037] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.156055 4897 raft_consensus.cc:3037] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.156931 4764 raft_consensus.cc:2454] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719b7b6d18b4ebe843fa62b5b7acab1 in term 1.
I20260924 00:09:50.156931 4897 raft_consensus.cc:2454] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9719b7b6d18b4ebe843fa62b5b7acab1 in term 1.
I20260924 00:09:50.157168 4567 leader_election.cc:302] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fa13ddfe8e4a12b5fdf601e667bc5f, 9719b7b6d18b4ebe843fa62b5b7acab1; no voters:
I20260924 00:09:50.157281 4967 raft_consensus.cc:2781] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:50.157469 4967 raft_consensus.cc:686] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 1 LEADER]: Becoming Leader. State: Replica: 9719b7b6d18b4ebe843fa62b5b7acab1, State: Running, Role: LEADER
I20260924 00:09:50.157601 4967 consensus_queue.cc:237] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } }
I20260924 00:09:50.158543 4487 catalog_manager.cc:5495] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9719b7b6d18b4ebe843fa62b5b7acab1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9719b7b6d18b4ebe843fa62b5b7acab1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:50.219861 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.217336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.0.98.1:44989)
0924 00:09:50.217382 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.217384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.217392 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:50.217433 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.217434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.217454 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.217533 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:50.217697 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.218164 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.218978 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:50.219174 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:50.219329 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:50.219523 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:50.219629 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:50.219728 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.219747 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.219770 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:50.221827 4897 raft_consensus.cc:1261] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 1 FOLLOWER]: Refusing update from remote peer 9719b7b6d18b4ebe843fa62b5b7acab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:50.221827 4764 raft_consensus.cc:1261] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f [term 1 FOLLOWER]: Refusing update from remote peer 9719b7b6d18b4ebe843fa62b5b7acab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:50.222210 4967 consensus_queue.cc:1035] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01fa13ddfe8e4a12b5fdf601e667bc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:50.222378 4971 consensus_queue.cc:1035] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2b127b9791a4aae943135c6bdb391ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:50.227401 4487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:50.227519 4487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:50.228276 4487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1f410d37eae415287c2220e03c0e163: Sending DeleteTablet for 3 replicas of tablet d8560ab60c7641e0ade83974e592bb10
I20260924 00:09:50.228602 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet d8560ab60c7641e0ade83974e592bb10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 00:09:50.228614 4877 tablet_service.cc:1514] Processing DeleteTablet for tablet d8560ab60c7641e0ade83974e592bb10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35382
I20260924 00:09:50.228670 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet d8560ab60c7641e0ade83974e592bb10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 00:09:50.228888 4986 tablet_replica.cc:330] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1: stopping tablet replica
I20260924 00:09:50.229030 4986 raft_consensus.cc:2229] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:50.229141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4549
I20260924 00:09:50.229300 4986 raft_consensus.cc:2258] T d8560ab60c7641e0ade83974e592bb10 P 9719b7b6d18b4ebe843fa62b5b7acab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:50.229725 4985 tablet_replica.cc:330] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: stopping tablet replica
I20260924 00:09:50.229830 4985 raft_consensus.cc:2229] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:50.229947 4985 raft_consensus.cc:2258] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:50.230389 4985 ts_tablet_manager.cc:1905] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:50.232412 4985 ts_tablet_manager.cc:1918] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:50.232492 4985 log.cc:1196] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/d8560ab60c7641e0ade83974e592bb10
I20260924 00:09:50.232750 4985 ts_tablet_manager.cc:1939] T d8560ab60c7641e0ade83974e592bb10 P f2b127b9791a4aae943135c6bdb391ac: Deleting consensus metadata
I20260924 00:09:50.233194 4471 catalog_manager.cc:4841] TS f2b127b9791a4aae943135c6bdb391ac (127.0.98.3:40613): tablet d8560ab60c7641e0ade83974e592bb10 (table test-table [id=7a209b895fa3451abb2d050bdb9c4625]) successfully deleted
W20260924 00:09:50.234963 4472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44989 (error 108)
I20260924 00:09:50.235260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4682
I20260924 00:09:50.235272 4984 tablet_replica.cc:330] T d8560ab60c7641e0ade83974e592bb10 P 01fa13ddfe8e4a12b5fdf601e667bc5f: stopping tablet replica
I20260924 00:09:50.235927 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.235409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44989 (local address 127.0.0.1:33792)
0924 00:09:50.235847 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.235886 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
W20260924 00:09:50.236311 4472 catalog_manager.cc:4584] TS 9719b7b6d18b4ebe843fa62b5b7acab1 (127.0.98.1:44989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8560ab60c7641e0ade83974e592bb10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44989: connect: Connection refused (error 111)
I20260924 00:09:50.240382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4815
I20260924 00:09:50.240442 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.240288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.0.1:44512)
0924 00:09:50.240389 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.240415 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 00:09:50.244585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4456
2026-09-24T00:09:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2174 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T00:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:50Z Disabled control of system clock
I20260924 00:09:50.272424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36235
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:09:50.352370 4991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:50.352577 4991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:50.352599 4991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:50.353775 4991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:50.353819 4991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:50.353832 4991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:50.353845 4991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:50.353907 4991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:50.355324 4991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:50.355599 4991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:50.355906 4991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:50.360829 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.360834 4996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.360872 4997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.361146 4991 server_base.cc:1034] running on GCE node
I20260924 00:09:50.361409 4991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:50.361716 4991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:50.362895 4991 hybrid_clock.cc:648] HybridClock initialized: now 1790208590362885 us; error 49 us; skew 500 ppm
I20260924 00:09:50.364292 4991 webserver.cc:460] Webserver started at http://127.0.98.62:44211/ using document root <none> and password file <none>
I20260924 00:09:50.364557 4991 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:50.364636 4991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:50.364787 4991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:50.365875 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "aa28202fb2c245418d7702392795c33d"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4b44d393050bb755a7d16efeab409a7f"
server_key_iv: "74bf0386fcb59dd6d49555b2b87e3fb8"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.366256 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa28202fb2c245418d7702392795c33d"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4b44d393050bb755a7d16efeab409a7f"
server_key_iv: "74bf0386fcb59dd6d49555b2b87e3fb8"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.367650 4991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:09:50.368427 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.368600 4991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:50.368708 4991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "aa28202fb2c245418d7702392795c33d"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4b44d393050bb755a7d16efeab409a7f"
server_key_iv: "74bf0386fcb59dd6d49555b2b87e3fb8"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.368789 4991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:50.372161 4991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:50.372442 4991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:50.372589 4991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:50.376339 4991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36235
I20260924 00:09:50.376400 5057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36235 every 8 connection(s)
I20260924 00:09:50.376745 4991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:50.377465 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:50.380290 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: Bootstrap starting.
I20260924 00:09:50.380837 5058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:50.381067 5058 log.cc:824] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:50.381724 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: No bootstrap required, opened a new log
I20260924 00:09:50.383312 5058 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260924 00:09:50.383451 5058 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:50.383473 5058 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa28202fb2c245418d7702392795c33d, State: Initialized, Role: FOLLOWER
I20260924 00:09:50.383610 5058 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260924 00:09:50.383683 5058 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:50.383713 5058 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:50.383740 5058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.384207 5058 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260924 00:09:50.384290 5058 leader_election.cc:302] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa28202fb2c245418d7702392795c33d; no voters:
I20260924 00:09:50.384419 5058 leader_election.cc:288] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:50.384505 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:50.384660 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [term 1 LEADER]: Becoming Leader. State: Replica: aa28202fb2c245418d7702392795c33d, State: Running, Role: LEADER
I20260924 00:09:50.384671 5058 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:50.384761 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } }
I20260924 00:09:50.385088 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa28202fb2c245418d7702392795c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } } }
I20260924 00:09:50.385175 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:50.385317 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa28202fb2c245418d7702392795c33d. Latest consensus state: current_term: 1 leader_uuid: "aa28202fb2c245418d7702392795c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa28202fb2c245418d7702392795c33d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36235 } } }
I20260924 00:09:50.385381 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:50.385913 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:50.386471 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:50.386528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 4991
I20260924 00:09:50.386617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:50.386806 392 external_mini_cluster.cc:1428] Setting key 616ef9b92f219d7f8dfb44d4816ab055
I20260924 00:09:50.388084 5068 catalog_manager.cc:1347] Generated new cluster ID: 77e5cf4b098146cfa87a2bb32034aea8
I20260924 00:09:50.388139 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:50.389417 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.387581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.98.62:36235)
0924 00:09:50.387795 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.387803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.387845 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:09:50.387958 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.387966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.388010 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.388158 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.388931 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.388934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.388935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.388938 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.388981 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.388994 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.389059 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.389171 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.389241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260924 00:09:50.397709 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:50.397914 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:50.405419 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: Generated new TSK 0
I20260924 00:09:50.405651 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:09:50.408007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:09:50.493876 5082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:50.494081 5082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:50.494104 5082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:50.495323 5082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:50.495374 5082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:50.495401 5082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:50.496768 5082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:50.497021 5082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:50.497340 5082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:50.503500 5087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.503510 5090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.503536 5088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.503836 5082 server_base.cc:1034] running on GCE node
I20260924 00:09:50.504045 5082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:50.504314 5082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:50.505620 5082 hybrid_clock.cc:648] HybridClock initialized: now 1790208590505598 us; error 36 us; skew 500 ppm
I20260924 00:09:50.507115 5082 webserver.cc:460] Webserver started at http://127.0.98.1:44615/ using document root <none> and password file <none>
I20260924 00:09:50.507418 5082 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:50.507496 5082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:50.507654 5082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:50.508762 5082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "2389ca583d994be997fa6fabf44ecb1f"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4602a0a97c8deaeb9036d1f03f609c1c"
server_key_iv: "c9d0e3cb1a4abf7a97bc96b11e4c9f85"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.509147 5082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2389ca583d994be997fa6fabf44ecb1f"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4602a0a97c8deaeb9036d1f03f609c1c"
server_key_iv: "c9d0e3cb1a4abf7a97bc96b11e4c9f85"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.510547 5082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:50.511356 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.511514 5082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:50.511624 5082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "2389ca583d994be997fa6fabf44ecb1f"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "4602a0a97c8deaeb9036d1f03f609c1c"
server_key_iv: "c9d0e3cb1a4abf7a97bc96b11e4c9f85"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.511698 5082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:50.515009 5082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:50.515378 5082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:50.515527 5082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:50.515779 5082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:50.516186 5082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:50.516229 5082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.516250 5082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:50.516295 5082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.523260 5082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41945
I20260924 00:09:50.523290 5209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41945 every 8 connection(s)
I20260924 00:09:50.523676 5082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:50.526818 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.524506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35143 (local address 127.0.98.62:36235)
0924 00:09:50.524582 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.524584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.524818 (+ 234us) server_negotiation.cc:407] Connection header received
0924 00:09:50.525308 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.525309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.525315 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.525385 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.526619 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.526620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.526620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.526624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.526635 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.526639 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.526686 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.526748 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.526761 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:50.527035 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.524497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.1:35143)
0924 00:09:50.524669 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.524713 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.525204 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.525442 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.525445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.525453 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.526473 (+ 1020us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.526473 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.526504 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.526534 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.526683 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.526686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.526689 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.526745 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.526796 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 00:09:50.527536 5210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260924 00:09:50.527623 5210 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:50.527812 5210 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260924 00:09:50.528390 5022 ts_manager.cc:194] Registered new tserver with Master: 2389ca583d994be997fa6fabf44ecb1f (127.0.98.1:41945)
I20260924 00:09:50.533365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5082
I20260924 00:09:50.533454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:50.533589 392 external_mini_cluster.cc:1428] Setting key 6c288a8356a7c0c1ba1cfbda154ab636
I20260924 00:09:50.534467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:09:50.617352 5213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:50.617573 5213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:50.617597 5213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:50.618791 5213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:50.618834 5213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:50.618860 5213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:50.620195 5213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:50.620399 5213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:50.620646 5213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:50.625923 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.626017 5213 server_base.cc:1034] running on GCE node
W20260924 00:09:50.625917 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.625917 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.626340 5213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:50.626556 5213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:50.627705 5213 hybrid_clock.cc:648] HybridClock initialized: now 1790208590627683 us; error 53 us; skew 500 ppm
I20260924 00:09:50.633162 5213 webserver.cc:460] Webserver started at http://127.0.98.2:45113/ using document root <none> and password file <none>
I20260924 00:09:50.633455 5213 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:50.633509 5213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:50.633685 5213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:50.634804 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "758369fc49e04cbea21a118d8be8d5a9"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "38922638252724991990eb8e06a43193"
server_key_iv: "000e7207d57fc2137811a433686daf0f"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.635167 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "758369fc49e04cbea21a118d8be8d5a9"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "38922638252724991990eb8e06a43193"
server_key_iv: "000e7207d57fc2137811a433686daf0f"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.636533 5213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:50.637275 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.637483 5213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:50.637588 5213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "758369fc49e04cbea21a118d8be8d5a9"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "38922638252724991990eb8e06a43193"
server_key_iv: "000e7207d57fc2137811a433686daf0f"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.637669 5213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:50.641222 5213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:50.641500 5213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:50.641642 5213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:50.641845 5213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:50.642143 5213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:50.642199 5213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.642237 5213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:50.642266 5213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.648243 5213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37717
I20260924 00:09:50.648286 5340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37717 every 8 connection(s)
I20260924 00:09:50.648607 5213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:50.649190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5213
I20260924 00:09:50.649313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:50.649502 392 external_mini_cluster.cc:1428] Setting key 12b80c120f0d0eb333bac1a42c8e1bb9
I20260924 00:09:50.650553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 00:09:50.651751 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.649303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42179 (local address 127.0.98.62:36235)
0924 00:09:50.649356 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.649358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.649631 (+ 273us) server_negotiation.cc:407] Connection header received
0924 00:09:50.650113 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.650114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.650119 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.650181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.651467 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.651469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.651469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.651474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.651493 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.651498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.651565 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.651640 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.651655 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:09:50.651975 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.649318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.2:42179)
0924 00:09:50.649504 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.649544 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.650033 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.650234 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.650237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.650245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.651309 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.651310 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.651340 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.651369 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.651578 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.651584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.651587 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.651649 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.651699 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 00:09:50.652441 5341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260924 00:09:50.652568 5341 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:50.652717 5341 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260924 00:09:50.653113 5022 ts_manager.cc:194] Registered new tserver with Master: 758369fc49e04cbea21a118d8be8d5a9 (127.0.98.2:37717)
W20260924 00:09:50.732746 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:50.732954 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:50.732975 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:50.734124 5344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:50.734167 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:50.734193 5344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:50.735637 5344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:50.735898 5344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:50.736202 5344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:50.741173 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.741148 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.741158 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.741470 5344 server_base.cc:1034] running on GCE node
I20260924 00:09:50.741712 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:50.741973 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:50.743141 5344 hybrid_clock.cc:648] HybridClock initialized: now 1790208590743117 us; error 37 us; skew 500 ppm
I20260924 00:09:50.744359 5344 webserver.cc:460] Webserver started at http://127.0.98.3:40919/ using document root <none> and password file <none>
I20260924 00:09:50.744654 5344 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:50.744732 5344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:50.744886 5344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:50.746014 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "b58f82a756bf4985903976c466c8d215"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "2869b056d4684dcf6c97432bba93b91d"
server_key_iv: "4462a6302d880f48fc88e25f6ed981fb"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.746416 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b58f82a756bf4985903976c466c8d215"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "2869b056d4684dcf6c97432bba93b91d"
server_key_iv: "4462a6302d880f48fc88e25f6ed981fb"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.747839 5344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:50.748605 5358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.748805 5344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:50.748911 5344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "b58f82a756bf4985903976c466c8d215"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "2869b056d4684dcf6c97432bba93b91d"
server_key_iv: "4462a6302d880f48fc88e25f6ed981fb"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.748996 5344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:50.752374 5344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:50.752650 5344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:50.752795 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:50.753005 5344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:50.753311 5344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:50.753371 5344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.753409 5344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:50.753443 5344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.759151 5344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39647
I20260924 00:09:50.759195 5471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39647 every 8 connection(s)
I20260924 00:09:50.759563 5344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:50.762338 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.760347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37549 (local address 127.0.98.62:36235)
0924 00:09:50.760417 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.760419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.760621 (+ 202us) server_negotiation.cc:407] Connection header received
0924 00:09:50.761050 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.761051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.761055 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.761114 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.762148 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.762149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.762149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.762153 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.762165 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.762169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.762215 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.762274 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.762287 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:09:50.762593 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.760323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36235 (local address 127.0.98.3:37549)
0924 00:09:50.760493 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.760533 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.760969 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.761164 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.761167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.761174 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.762015 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.762016 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.762041 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.762066 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.762213 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.762215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.762217 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.762266 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.762313 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260924 00:09:50.763119 5472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36235
I20260924 00:09:50.763226 5472 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:50.763405 5472 heartbeater.cc:511] Master 127.0.98.62:36235 requested a full tablet report, sending...
I20260924 00:09:50.763777 5022 ts_manager.cc:194] Registered new tserver with Master: b58f82a756bf4985903976c466c8d215 (127.0.98.3:39647)
I20260924 00:09:50.765857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5344
I20260924 00:09:50.765964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:50.766155 392 external_mini_cluster.cc:1428] Setting key 02439a7cfe4267e546bd690190b99337
I20260924 00:09:50.766882 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:09:50.769351 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.768284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.62:36235)
0924 00:09:50.768344 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.768346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.768467 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:09:50.768507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.768507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.768511 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.768564 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.769195 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.769196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.769197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.769200 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.769213 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.769216 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.769263 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.769295 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.769306 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:09:50.770960 5022 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:50.772737 5022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:50.777073 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.774975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41945 (local address 127.0.0.1:44968)
0924 00:09:50.775416 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.775429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.775485 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.776061 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.776067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.776074 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.776624 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.776625 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.776637 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.776655 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.776897 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.776903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.776906 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.776978 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.776992 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260924 00:09:50.777112 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.775163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.1:41945)
0924 00:09:50.775579 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.775589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.775597 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:50.775643 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.775646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.775655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.775732 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.776736 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.776743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.776744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.776748 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.776781 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.776805 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.776901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.776995 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.777014 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":150,"threads_started":1}
I20260924 00:09:50.777331 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.774975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37717 (local address 127.0.0.1:36992)
0924 00:09:50.775373 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.775411 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.775498 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.776283 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.776283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.776285 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.776803 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.776804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.776810 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.776824 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.777084 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.777084 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.777085 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.777266 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.777279 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":77,"threads_started":1}
I20260924 00:09:50.777541 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.775098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.0.98.2:37717)
0924 00:09:50.775346 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.775353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.775498 (+ 145us) server_negotiation.cc:407] Connection header received
0924 00:09:50.775660 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.775667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.775679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.775749 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.776913 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.776920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.776921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.776925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.776973 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.776998 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.777058 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.777447 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.777464 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260924 00:09:50.777670 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.775409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.0.98.3:39647)
0924 00:09:50.776099 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.776103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.776234 (+ 131us) server_negotiation.cc:407] Connection header received
0924 00:09:50.776298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.776300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.776312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.776384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.777053 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.777058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.777059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.777063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.777096 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.777120 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.777424 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.777556 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.777572 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":70,"threads_started":1}
I20260924 00:09:50.777827 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.775187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39647 (local address 127.0.0.1:43014)
0924 00:09:50.776146 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.776156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.776217 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.776408 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.776409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.776411 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.776953 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.776954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.776960 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.776971 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.777479 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.777480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.777481 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.777754 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.777770 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":124,"threads_started":1}
I20260924 00:09:50.777854 5144 tablet_service.cc:1467] Processing CreateTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 (DEFAULT_TABLE table=test-table [id=03ba09abf9f14875bb0423996944df5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:50.778244 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e41dde9e434df7a2c0a18ff0868f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:50.778476 5406 tablet_service.cc:1467] Processing CreateTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 (DEFAULT_TABLE table=test-table [id=03ba09abf9f14875bb0423996944df5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:50.778761 5406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e41dde9e434df7a2c0a18ff0868f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:50.779852 5275 tablet_service.cc:1467] Processing CreateTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 (DEFAULT_TABLE table=test-table [id=03ba09abf9f14875bb0423996944df5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:50.780179 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e41dde9e434df7a2c0a18ff0868f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:50.781301 5486 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: Bootstrap starting.
I20260924 00:09:50.781852 5487 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Bootstrap starting.
I20260924 00:09:50.782470 5486 tablet_bootstrap.cc:654] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:50.782481 5487 tablet_bootstrap.cc:654] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:50.782814 5486 log.cc:824] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:50.782814 5487 log.cc:824] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:50.783320 5490 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Bootstrap starting.
I20260924 00:09:50.783602 5486 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: No bootstrap required, opened a new log
I20260924 00:09:50.783649 5486 ts_tablet_manager.cc:1397] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:50.783640 5487 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: No bootstrap required, opened a new log
I20260924 00:09:50.783705 5487 ts_tablet_manager.cc:1397] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:50.783998 5490 tablet_bootstrap.cc:654] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:50.784332 5490 log.cc:824] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:50.785152 5490 tablet_bootstrap.cc:492] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: No bootstrap required, opened a new log
I20260924 00:09:50.785211 5490 ts_tablet_manager.cc:1397] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:50.785867 5486 raft_consensus.cc:348] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.786026 5486 raft_consensus.cc:374] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:50.786049 5486 raft_consensus.cc:729] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2389ca583d994be997fa6fabf44ecb1f, State: Initialized, Role: FOLLOWER
I20260924 00:09:50.786162 5486 consensus_queue.cc:260] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.786414 5486 ts_tablet_manager.cc:1428] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:09:50.786432 5210 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260924 00:09:50.786432 5487 raft_consensus.cc:348] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.786599 5487 raft_consensus.cc:374] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:50.786635 5487 raft_consensus.cc:729] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58f82a756bf4985903976c466c8d215, State: Initialized, Role: FOLLOWER
I20260924 00:09:50.786784 5487 consensus_queue.cc:260] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.787025 5487 ts_tablet_manager.cc:1428] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:09:50.787132 5472 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260924 00:09:50.787942 5490 raft_consensus.cc:348] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.788067 5490 raft_consensus.cc:374] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:50.788089 5490 raft_consensus.cc:729] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758369fc49e04cbea21a118d8be8d5a9, State: Initialized, Role: FOLLOWER
I20260924 00:09:50.788203 5490 consensus_queue.cc:260] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.788407 5490 ts_tablet_manager.cc:1428] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:09:50.788494 5341 heartbeater.cc:503] Master 127.0.98.62:36235 was elected leader, sending a full tablet report...
I20260924 00:09:50.808117 5493 raft_consensus.cc:482] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:50.808235 5493 raft_consensus.cc:504] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.808586 5493 leader_election.cc:288] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 758369fc49e04cbea21a118d8be8d5a9 (127.0.98.2:37717), 2389ca583d994be997fa6fabf44ecb1f (127.0.98.1:41945)
I20260924 00:09:50.809981 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.808755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37717 (local address 127.0.98.3:55149)
0924 00:09:50.808820 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.808832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.808871 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.809052 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.809053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.809056 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.809569 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.809569 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.809577 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.809591 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.809821 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.809822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.809823 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.809883 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.809897 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:09:50.810026 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.808839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55149 (local address 127.0.98.2:37717)
0924 00:09:50.808887 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.808889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.808899 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:50.808926 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.808927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.808934 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.809004 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.809693 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.809694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.809695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.809699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.809716 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.809722 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.809788 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.809903 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.809934 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:50.810093 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.808885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55751 (local address 127.0.98.1:41945)
0924 00:09:50.808929 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.808931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.809038 (+ 107us) server_negotiation.cc:407] Connection header received
0924 00:09:50.809067 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.809068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.809074 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.809137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.809855 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.809856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.809857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.809862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.809880 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.809884 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.809940 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.810002 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.810015 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:50.810108 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.808767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41945 (local address 127.0.98.3:55751)
0924 00:09:50.808957 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.808966 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:50.809016 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:50.809151 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:50.809152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:50.809154 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:50.809748 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:09:50.809749 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:50.809755 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:09:50.809768 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:50.809949 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:50.809950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:50.809951 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:09:50.810003 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:09:50.810015 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":125,"threads_started":1}
I20260924 00:09:50.810388 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e41dde9e434df7a2c0a18ff0868f01" candidate_uuid: "b58f82a756bf4985903976c466c8d215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2389ca583d994be997fa6fabf44ecb1f" is_pre_election: true
I20260924 00:09:50.810355 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e41dde9e434df7a2c0a18ff0868f01" candidate_uuid: "b58f82a756bf4985903976c466c8d215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758369fc49e04cbea21a118d8be8d5a9" is_pre_election: true
I20260924 00:09:50.810542 5295 raft_consensus.cc:2454] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b58f82a756bf4985903976c466c8d215 in term 0.
I20260924 00:09:50.810542 5164 raft_consensus.cc:2454] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b58f82a756bf4985903976c466c8d215 in term 0.
I20260924 00:09:50.810770 5359 leader_election.cc:302] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2389ca583d994be997fa6fabf44ecb1f, b58f82a756bf4985903976c466c8d215; no voters:
I20260924 00:09:50.810904 5493 raft_consensus.cc:2781] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:50.810956 5493 raft_consensus.cc:482] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:50.810981 5493 raft_consensus.cc:3037] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.811744 5493 raft_consensus.cc:504] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.811858 5493 leader_election.cc:288] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [CANDIDATE]: Term 1 election: Requested vote from peers 758369fc49e04cbea21a118d8be8d5a9 (127.0.98.2:37717), 2389ca583d994be997fa6fabf44ecb1f (127.0.98.1:41945)
I20260924 00:09:50.811996 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e41dde9e434df7a2c0a18ff0868f01" candidate_uuid: "b58f82a756bf4985903976c466c8d215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758369fc49e04cbea21a118d8be8d5a9"
I20260924 00:09:50.812047 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e41dde9e434df7a2c0a18ff0868f01" candidate_uuid: "b58f82a756bf4985903976c466c8d215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2389ca583d994be997fa6fabf44ecb1f"
I20260924 00:09:50.812074 5295 raft_consensus.cc:3037] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.812104 5164 raft_consensus.cc:3037] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:50.812709 5295 raft_consensus.cc:2454] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b58f82a756bf4985903976c466c8d215 in term 1.
I20260924 00:09:50.812716 5164 raft_consensus.cc:2454] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b58f82a756bf4985903976c466c8d215 in term 1.
I20260924 00:09:50.812889 5362 leader_election.cc:302] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758369fc49e04cbea21a118d8be8d5a9, b58f82a756bf4985903976c466c8d215; no voters:
I20260924 00:09:50.812976 5493 raft_consensus.cc:2781] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:50.813113 5493 raft_consensus.cc:686] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 1 LEADER]: Becoming Leader. State: Replica: b58f82a756bf4985903976c466c8d215, State: Running, Role: LEADER
I20260924 00:09:50.813215 5493 consensus_queue.cc:237] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } }
I20260924 00:09:50.814101 5022 catalog_manager.cc:5495] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 reported cstate change: term changed from 0 to 1, leader changed from <none> to b58f82a756bf4985903976c466c8d215 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b58f82a756bf4985903976c466c8d215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b58f82a756bf4985903976c466c8d215" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39647 } health_report { overall_health: HEALTHY } } }
I20260924 00:09:50.827950 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.826732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.0.98.3:39647)
0924 00:09:50.826798 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:50.826799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:50.826807 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:50.826841 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:50.826841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:50.826853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:50.826903 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:50.827614 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:50.827615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:50.827615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:50.827619 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:50.827673 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:50.827680 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:50.827796 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:50.827846 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 00:09:50.827859 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:09:50.829722 5295 raft_consensus.cc:1261] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 1 FOLLOWER]: Refusing update from remote peer b58f82a756bf4985903976c466c8d215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:50.829754 5164 raft_consensus.cc:1261] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 1 FOLLOWER]: Refusing update from remote peer b58f82a756bf4985903976c466c8d215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:50.830031 5493 consensus_queue.cc:1035] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "758369fc49e04cbea21a118d8be8d5a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:50.830176 5496 consensus_queue.cc:1035] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2389ca583d994be997fa6fabf44ecb1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:50.832877 5500 mvcc.cc:205] Tried to move back new op lower bound from 7332694388035960832 to 7332694387971571712. Current Snapshot: MvccSnapshot[applied={T|T < 7332694388035960832}]
I20260924 00:09:50.835069 5021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:50.835151 5021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:50.836050 5021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa28202fb2c245418d7702392795c33d: Sending DeleteTablet for 3 replicas of tablet b8e41dde9e434df7a2c0a18ff0868f01
I20260924 00:09:50.836410 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='slave'} at 127.0.0.1:44968
I20260924 00:09:50.836475 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='slave'} at 127.0.0.1:36992
I20260924 00:09:50.836457 5406 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e41dde9e434df7a2c0a18ff0868f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:50 UTC) from {username='slave'} at 127.0.0.1:43014
I20260924 00:09:50.836661 5509 tablet_replica.cc:330] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f: stopping tablet replica
I20260924 00:09:50.836748 5509 raft_consensus.cc:2229] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:50.836822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5082
I20260924 00:09:50.836853 5509 raft_consensus.cc:2258] T b8e41dde9e434df7a2c0a18ff0868f01 P 2389ca583d994be997fa6fabf44ecb1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:50.837402 5510 tablet_replica.cc:330] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: stopping tablet replica
I20260924 00:09:50.837538 5510 raft_consensus.cc:2229] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:50.837637 5510 raft_consensus.cc:2258] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:50.838063 5510 ts_tablet_manager.cc:1905] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:50.840031 5510 ts_tablet_manager.cc:1918] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:50.840109 5510 log.cc:1196] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/b8e41dde9e434df7a2c0a18ff0868f01
I20260924 00:09:50.840348 5510 ts_tablet_manager.cc:1939] T b8e41dde9e434df7a2c0a18ff0868f01 P 758369fc49e04cbea21a118d8be8d5a9: Deleting consensus metadata
I20260924 00:09:50.840866 5009 catalog_manager.cc:4841] TS 758369fc49e04cbea21a118d8be8d5a9 (127.0.98.2:37717): tablet b8e41dde9e434df7a2c0a18ff0868f01 (table test-table [id=03ba09abf9f14875bb0423996944df5c]) successfully deleted
W20260924 00:09:50.842983 5006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41945 (error 108)
I20260924 00:09:50.843236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5213
I20260924 00:09:50.843278 5511 tablet_replica.cc:330] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: stopping tablet replica
I20260924 00:09:50.843379 5511 raft_consensus.cc:2229] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:50.843494 5511 raft_consensus.cc:2258] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:50.845148 5511 ts_tablet_manager.cc:1905] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:50.845191 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:50.845068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41945 (local address 127.0.0.1:44974)
0924 00:09:50.845114 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:50.845154 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 00:09:50.845294 5006 catalog_manager.cc:4584] TS 2389ca583d994be997fa6fabf44ecb1f (127.0.98.1:41945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8e41dde9e434df7a2c0a18ff0868f01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41945: connect: Connection refused (error 111)
I20260924 00:09:50.846966 5511 ts_tablet_manager.cc:1918] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:50.847047 5511 log.cc:1196] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/b8e41dde9e434df7a2c0a18ff0868f01
I20260924 00:09:50.847395 5511 ts_tablet_manager.cc:1939] T b8e41dde9e434df7a2c0a18ff0868f01 P b58f82a756bf4985903976c466c8d215: Deleting consensus metadata
I20260924 00:09:50.847896 5009 catalog_manager.cc:4841] TS b58f82a756bf4985903976c466c8d215 (127.0.98.3:39647): tablet b8e41dde9e434df7a2c0a18ff0868f01 (table test-table [id=03ba09abf9f14875bb0423996944df5c]) successfully deleted
I20260924 00:09:50.849020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5344
I20260924 00:09:50.854417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 4991
2026-09-24T00:09:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (608 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 00:09:50.873728 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:09:50.874816 392 webserver.cc:460] Webserver started at https://127.0.0.1:38357/ using document root <none> and password file <none>
I20260924 00:09:50.874897 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:09:50.875095 392 webserver.cc:460] Webserver started at http://127.0.0.1:40021/ using document root <none> and password file <none>
2026-09-24T00:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:50Z Disabled control of system clock
I20260924 00:09:50.883183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/testchainca.cert with env {}
W20260924 00:09:50.965529 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:50.965729 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:50.965750 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:50.966962 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:50.967031 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:50.967046 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:50.967072 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:50.967144 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:09:50.967161 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:09:50.967176 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38357/jwks.json
I20260924 00:09:50.968730 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:50.969007 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:50.969238 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:50.974108 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.974116 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:50.974129 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:50.974316 5518 server_base.cc:1034] running on GCE node
I20260924 00:09:50.974658 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:50.974970 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:50.976140 5518 hybrid_clock.cc:648] HybridClock initialized: now 1790208590976126 us; error 47 us; skew 500 ppm
I20260924 00:09:50.977340 5518 webserver.cc:460] Webserver started at http://127.0.98.62:39861/ using document root <none> and password file <none>
I20260924 00:09:50.977591 5518 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:50.977659 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:50.977805 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:50.978849 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "c347c84b99234490956764ccd745cf32"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "85bd6b2fd691fffe74172ad4a36e7d76"
server_key_iv: "890d797dcdf2b0d107a2ec3a9ebb8d30"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.979255 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "c347c84b99234490956764ccd745cf32"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "85bd6b2fd691fffe74172ad4a36e7d76"
server_key_iv: "890d797dcdf2b0d107a2ec3a9ebb8d30"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.980576 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:50.981426 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:50.981642 5518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:50.981765 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "c347c84b99234490956764ccd745cf32"
format_stamp: "Formatted at 2026-09-24 00:09:50 on dist-test-slave-3q8k"
server_key: "85bd6b2fd691fffe74172ad4a36e7d76"
server_key_iv: "890d797dcdf2b0d107a2ec3a9ebb8d30"
server_key_version: "encryptionkey@0"
I20260924 00:09:50.981856 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:50.999143 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:50.999514 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:50.999689 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:51.003928 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35583
I20260924 00:09:51.003970 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35583 every 8 connection(s)
I20260924 00:09:51.004343 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:51.005321 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:51.007678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5518
I20260924 00:09:51.007794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:51.007915 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32: Bootstrap starting.
I20260924 00:09:51.007969 392 external_mini_cluster.cc:1428] Setting key af974105fcbbd5d45e3d00fe8944575c
I20260924 00:09:51.008613 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:51.008931 5585 log.cc:824] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:51.009632 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32: No bootstrap required, opened a new log
I20260924 00:09:51.011459 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260924 00:09:51.011623 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:51.011670 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c347c84b99234490956764ccd745cf32, State: Initialized, Role: FOLLOWER
I20260924 00:09:51.011797 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260924 00:09:51.011888 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:51.011930 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:51.011978 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:51.012512 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260924 00:09:51.012646 5585 leader_election.cc:302] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c347c84b99234490956764ccd745cf32; no voters:
I20260924 00:09:51.012807 5585 leader_election.cc:288] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:51.012954 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:51.013069 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [term 1 LEADER]: Becoming Leader. State: Replica: c347c84b99234490956764ccd745cf32, State: Running, Role: LEADER
I20260924 00:09:51.013118 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:51.013151 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260924 00:09:51.013626 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c347c84b99234490956764ccd745cf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260924 00:09:51.013733 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.013633 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c347c84b99234490956764ccd745cf32. Latest consensus state: current_term: 1 leader_uuid: "c347c84b99234490956764ccd745cf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c347c84b99234490956764ccd745cf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260924 00:09:51.013935 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.014091 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.008490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.0.98.62:35583)
0924 00:09:51.008729 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.008739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.008772 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:09:51.009270 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.009277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.009340 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.009520 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:51.009753 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.010629 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.011550 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.012159 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.012309 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.013158 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:51.013162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:51.013162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:51.013354 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:51.013395 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:51.013408 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:51.013447 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:51.013684 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 00:09:51.013768 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260924 00:09:51.014346 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:51.014945 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:51.015190 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:09:51.016631 5599 catalog_manager.cc:1347] Generated new cluster ID: ff634282fcfe49b9a0cb3061b48f5566
I20260924 00:09:51.016674 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:51.031630 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:51.032164 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:51.038717 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c347c84b99234490956764ccd745cf32: Generated new TSK 0
I20260924 00:09:51.038960 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:09:51.060114 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 00:09:51.060225 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:09:51.060482 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.044794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.98.62:35583)
0924 00:09:51.044839 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.044842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.044961 (+ 119us) server_negotiation.cc:407] Connection header received
0924 00:09:51.045064 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.045065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.045077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.045142 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:09:51.045305 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.045831 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.046676 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.046882 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.046972 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.047237 (+ 265us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:09:51.060246 (+ 13009us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:09:51.060401 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:09:51.060559 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 00:09:51.060889 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:51.044699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35583 (local address 127.0.0.1:59918)
0924 00:09:51.044879 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:51.044890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:51.044982 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:51.045147 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:51.045148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:51.045150 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:09:51.045220 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.045222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.045935 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.045936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.046537 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.046540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.046972 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.046976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.047040 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.047064 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:09:51.060430 (+ 13366us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:09:51.060528 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 00:09:51.061339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5518
2026-09-24T00:09:51Z chronyd exiting
I20260924 00:09:51.097214 392 test_util.cc:175] -----------------------------------------------
I20260924 00:09:51.097291 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790208568194258-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 00:09:51.098094 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:09:51.098788 392 webserver.cc:460] Webserver started at https://127.0.0.1:33329/ using document root <none> and password file <none>
I20260924 00:09:51.098868 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:09:51.099046 392 webserver.cc:460] Webserver started at http://127.0.0.1:43367/ using document root <none> and password file <none>
2026-09-24T00:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:51Z Disabled control of system clock
I20260924 00:09:51.106645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44721
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33329/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/testchainca.cert with env {}
W20260924 00:09:51.186422 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:51.186625 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:51.186646 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:51.187879 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:51.187922 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:51.187934 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:51.187947 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:51.188001 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:09:51.188022 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:09:51.188050 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33329/jwks.json
I20260924 00:09:51.189535 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33329/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:51.189796 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:51.190039 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:51.194854 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:51.194866 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:51.194883 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:51.195281 5621 server_base.cc:1034] running on GCE node
I20260924 00:09:51.195556 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:51.195845 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:51.196993 5621 hybrid_clock.cc:648] HybridClock initialized: now 1790208591196981 us; error 32 us; skew 500 ppm
I20260924 00:09:51.198269 5621 webserver.cc:460] Webserver started at http://127.0.98.62:41017/ using document root <none> and password file <none>
I20260924 00:09:51.198553 5621 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:51.198624 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:51.198794 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:51.199966 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "544ad9d7f8944b93a690d3fc2e509724"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "4b77e055d52c8255e2fb01769b59a237"
server_key_iv: "382da686d6550068de178d04abf83060"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.200335 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "544ad9d7f8944b93a690d3fc2e509724"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "4b77e055d52c8255e2fb01769b59a237"
server_key_iv: "382da686d6550068de178d04abf83060"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.201689 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:09:51.202437 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:51.202613 5621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:51.202709 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "544ad9d7f8944b93a690d3fc2e509724"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "4b77e055d52c8255e2fb01769b59a237"
server_key_iv: "382da686d6550068de178d04abf83060"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.202795 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:51.217796 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:51.218147 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:51.218324 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:51.222471 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44721
I20260924 00:09:51.222525 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44721 every 8 connection(s)
I20260924 00:09:51.222896 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:51.223918 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:51.226132 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724: Bootstrap starting.
I20260924 00:09:51.226725 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:51.227018 5688 log.cc:824] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:51.227756 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724: No bootstrap required, opened a new log
I20260924 00:09:51.229465 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } }
I20260924 00:09:51.229596 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:51.229619 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544ad9d7f8944b93a690d3fc2e509724, State: Initialized, Role: FOLLOWER
I20260924 00:09:51.229792 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } }
I20260924 00:09:51.229853 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:51.229876 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:51.229954 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:51.230487 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } }
I20260924 00:09:51.230571 5688 leader_election.cc:302] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 544ad9d7f8944b93a690d3fc2e509724; no voters:
I20260924 00:09:51.230741 5688 leader_election.cc:288] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:51.230890 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:51.231035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5621
I20260924 00:09:51.231057 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [term 1 LEADER]: Becoming Leader. State: Replica: 544ad9d7f8944b93a690d3fc2e509724, State: Running, Role: LEADER
I20260924 00:09:51.231040 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:51.231134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:51.231194 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } }
I20260924 00:09:51.231346 392 external_mini_cluster.cc:1428] Setting key 615dca7fff06a87fc8d12b5cb173881d
I20260924 00:09:51.231583 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544ad9d7f8944b93a690d3fc2e509724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } } }
I20260924 00:09:51.231663 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.231844 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544ad9d7f8944b93a690d3fc2e509724. Latest consensus state: current_term: 1 leader_uuid: "544ad9d7f8944b93a690d3fc2e509724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544ad9d7f8944b93a690d3fc2e509724" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44721 } } }
I20260924 00:09:51.231906 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.232180 5700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:51.232681 5700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:51.234851 5700 catalog_manager.cc:1347] Generated new cluster ID: 91353e233f664865aa7896efb7485a3b
I20260924 00:09:51.234905 5700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:51.237253 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.232004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.0.98.62:44721)
0924 00:09:51.232215 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.232224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.232310 (+ 86us) server_negotiation.cc:407] Connection header received
0924 00:09:51.232483 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.232492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.232565 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.232704 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:51.233135 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.234213 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.235274 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.235685 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.235835 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.236562 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:51.236566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:51.236566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:51.236742 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:51.236773 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:51.236783 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:51.236874 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:51.236940 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 00:09:51.237104 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260924 00:09:51.238135 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:09:51.284844 5700 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:51.285409 5700 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:51.290649 5700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544ad9d7f8944b93a690d3fc2e509724: Generated new TSK 0
I20260924 00:09:51.290849 5700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:09:51.317345 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 00:09:51.317445 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:09:51.317711 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.295841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.98.62:44721)
0924 00:09:51.295955 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.295958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.296092 (+ 134us) server_negotiation.cc:407] Connection header received
0924 00:09:51.296222 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.296224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.296237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.296314 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:09:51.296530 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.297330 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.298480 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.298667 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.298720 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.298932 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:09:51.317426 (+ 18494us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33329/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:09:51.317627 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33329/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 00:09:51.317783 5703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33329/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 00:09:51.317879 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:51.295750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44721 (local address 127.0.0.1:56072)
0924 00:09:51.295971 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:51.296022 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:51.296129 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:51.296346 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:51.296347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:51.296349 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:09:51.296430 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.296432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.297431 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.297432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.298390 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.298393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.298751 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.298755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.298821 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.298828 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:09:51.317638 (+ 18810us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33329/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:09:51.317765 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33329/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260924 00:09:51.318284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5621
2026-09-24T00:09:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 00:09:51.356189 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:09:51.356909 392 webserver.cc:460] Webserver started at https://127.0.0.1:36665/ using document root <none> and password file <none>
I20260924 00:09:51.356985 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:09:51.357173 392 webserver.cc:460] Webserver started at http://127.0.0.1:40927/ using document root <none> and password file <none>
2026-09-24T00:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:51Z Disabled control of system clock
I20260924 00:09:51.365046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39579
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39579
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36665/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:09:51.444341 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:51.444567 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:51.444589 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:51.445665 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:51.445699 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:51.445712 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:51.445724 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:51.445736 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:09:51.445748 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:09:51.445786 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36665/jwks.json
I20260924 00:09:51.447085 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36665/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:51.447376 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:51.447716 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:51.452778 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:51.452813 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:51.452893 5724 server_base.cc:1034] running on GCE node
W20260924 00:09:51.452795 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:51.453181 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:51.453433 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:51.454581 5724 hybrid_clock.cc:648] HybridClock initialized: now 1790208591454571 us; error 31 us; skew 500 ppm
I20260924 00:09:51.455859 5724 webserver.cc:460] Webserver started at http://127.0.98.62:44227/ using document root <none> and password file <none>
I20260924 00:09:51.456115 5724 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:51.456161 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:51.456337 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:51.457487 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "0731fbb5a4804d7db5d84491616c360b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "1e3dbeb5726dc982738cdcefd24d2b46"
server_key_iv: "3601b5c9affefca82a7040a433694103"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.457860 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "0731fbb5a4804d7db5d84491616c360b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "1e3dbeb5726dc982738cdcefd24d2b46"
server_key_iv: "3601b5c9affefca82a7040a433694103"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.459331 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:09:51.460144 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:51.460330 5724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:51.460430 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "0731fbb5a4804d7db5d84491616c360b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "1e3dbeb5726dc982738cdcefd24d2b46"
server_key_iv: "3601b5c9affefca82a7040a433694103"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.460511 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:51.476751 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:51.477097 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:51.477258 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:51.481433 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39579
I20260924 00:09:51.481470 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39579 every 8 connection(s)
I20260924 00:09:51.481822 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:51.482586 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:51.485314 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b: Bootstrap starting.
I20260924 00:09:51.485912 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:51.486204 5791 log.cc:824] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:51.486919 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b: No bootstrap required, opened a new log
I20260924 00:09:51.488648 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } }
I20260924 00:09:51.488790 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:51.488813 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0731fbb5a4804d7db5d84491616c360b, State: Initialized, Role: FOLLOWER
I20260924 00:09:51.489001 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } }
I20260924 00:09:51.489063 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:51.489127 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:51.489180 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:51.489679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5724
I20260924 00:09:51.489794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:51.489938 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } }
I20260924 00:09:51.489977 392 external_mini_cluster.cc:1428] Setting key 3417949f5847e3a859a6f6c5f867016c
I20260924 00:09:51.490051 5791 leader_election.cc:302] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0731fbb5a4804d7db5d84491616c360b; no voters:
I20260924 00:09:51.490262 5791 leader_election.cc:288] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:51.490286 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:51.490407 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [term 1 LEADER]: Becoming Leader. State: Replica: 0731fbb5a4804d7db5d84491616c360b, State: Running, Role: LEADER
I20260924 00:09:51.490478 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } }
I20260924 00:09:51.490538 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:51.490790 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0731fbb5a4804d7db5d84491616c360b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } } }
I20260924 00:09:51.490854 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.490969 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0731fbb5a4804d7db5d84491616c360b. Latest consensus state: current_term: 1 leader_uuid: "0731fbb5a4804d7db5d84491616c360b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0731fbb5a4804d7db5d84491616c360b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39579 } } }
I20260924 00:09:51.491037 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.491542 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:51.492081 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:51.494819 5800 catalog_manager.cc:1347] Generated new cluster ID: 6872743f4d77404fa5866d13f9aee01a
I20260924 00:09:51.494889 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:51.496778 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.490488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.0.98.62:39579)
0924 00:09:51.490718 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.490729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.490934 (+ 205us) server_negotiation.cc:407] Connection header received
0924 00:09:51.491266 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.491274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.491357 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.491495 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:51.491884 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.493337 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.494482 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.495059 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.495259 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.496010 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:51.496013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:51.496014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:51.496199 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:51.496232 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:51.496243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:51.496325 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:51.496409 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 00:09:51.496512 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 00:09:51.500204 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:09:51.514796 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:51.515465 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:51.519309 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0731fbb5a4804d7db5d84491616c360b: Generated new TSK 0
I20260924 00:09:51.519551 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:09:51.534585 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.530967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.98.62:39579)
0924 00:09:51.531029 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.531031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.531152 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:09:51.531300 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.531301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.531316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.531390 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:09:51.531540 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.532080 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.532920 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.533117 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.533210 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.533428 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:09:51.534377 (+ 949us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:09:51.534510 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 00:09:51.534649 5798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 00:09:51.534713 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:51.530876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39579 (local address 127.0.0.1:50980)
0924 00:09:51.531076 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:51.531086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:51.531172 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:51.531395 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:51.531396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:51.531398 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:09:51.531453 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.531454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.532188 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.532189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.532812 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.532814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.533211 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.533220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.533287 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.533292 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:09:51.534498 (+ 1206us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:09:51.534606 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39579: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260924 00:09:51.534945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5724
E20260924 00:09:51.545743 5821 webserver.cc:533] Webserver: sslize failed:
E20260924 00:09:51.545831 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T00:09:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T00:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:51Z Disabled control of system clock
I20260924 00:09:51.580717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35897
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35897
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/testchainca.cert with env {}
W20260924 00:09:51.663635 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:51.663916 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:51.663956 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:51.665184 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:51.665272 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:51.665297 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:51.665318 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:51.665344 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:09:51.665365 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:09:51.665385 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 00:09:51.667045 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:51.667372 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:51.667616 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:51.672971 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:51.672966 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:51.672978 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:51.673321 5825 server_base.cc:1034] running on GCE node
I20260924 00:09:51.673642 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:51.673946 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:51.675112 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790208591675101 us; error 29 us; skew 500 ppm
I20260924 00:09:51.676383 5825 webserver.cc:460] Webserver started at http://127.0.98.62:41471/ using document root <none> and password file <none>
I20260924 00:09:51.676643 5825 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:51.676716 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:51.676862 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:51.677956 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "81d4251900da4946a821ef4cfdac910b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "40fe41cd6242a7131cc525b78c91e67d"
server_key_iv: "8eeec923f720d035ab65dde7a30beead"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.678328 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "81d4251900da4946a821ef4cfdac910b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "40fe41cd6242a7131cc525b78c91e67d"
server_key_iv: "8eeec923f720d035ab65dde7a30beead"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.679761 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:51.680509 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:51.680691 5825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:51.680796 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "81d4251900da4946a821ef4cfdac910b"
format_stamp: "Formatted at 2026-09-24 00:09:51 on dist-test-slave-3q8k"
server_key: "40fe41cd6242a7131cc525b78c91e67d"
server_key_iv: "8eeec923f720d035ab65dde7a30beead"
server_key_version: "encryptionkey@0"
I20260924 00:09:51.680879 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:51.693874 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:51.694250 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:51.694432 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:51.698544 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35897
I20260924 00:09:51.698575 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35897 every 8 connection(s)
I20260924 00:09:51.698945 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:51.699945 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:51.702231 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b: Bootstrap starting.
I20260924 00:09:51.702827 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:51.703116 5892 log.cc:824] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:51.703845 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b: No bootstrap required, opened a new log
I20260924 00:09:51.705289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5825
I20260924 00:09:51.705403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:51.705590 392 external_mini_cluster.cc:1428] Setting key 6ad46be748688d3936ef0f9da6bbcc57
I20260924 00:09:51.705786 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } }
I20260924 00:09:51.705948 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:51.705993 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d4251900da4946a821ef4cfdac910b, State: Initialized, Role: FOLLOWER
I20260924 00:09:51.706125 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } }
I20260924 00:09:51.706218 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:51.706261 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:51.706312 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:51.706979 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } }
I20260924 00:09:51.707067 5892 leader_election.cc:302] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d4251900da4946a821ef4cfdac910b; no voters:
I20260924 00:09:51.707316 5892 leader_election.cc:288] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:51.707472 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:51.707581 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [term 1 LEADER]: Becoming Leader. State: Replica: 81d4251900da4946a821ef4cfdac910b, State: Running, Role: LEADER
I20260924 00:09:51.707616 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:51.707664 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } }
I20260924 00:09:51.707978 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d4251900da4946a821ef4cfdac910b. Latest consensus state: current_term: 1 leader_uuid: "81d4251900da4946a821ef4cfdac910b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } } }
I20260924 00:09:51.708063 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.708227 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d4251900da4946a821ef4cfdac910b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d4251900da4946a821ef4cfdac910b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35897 } } }
I20260924 00:09:51.708323 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:51.708474 5903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:51.709076 5903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:51.711040 5903 catalog_manager.cc:1347] Generated new cluster ID: b35fbb6bd9e84a8dafa8b82c33d0e176
I20260924 00:09:51.711098 5903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:51.711764 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.706075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.62:35897)
0924 00:09:51.706265 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.706274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.706312 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:09:51.706485 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.706491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.706543 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.706658 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:51.706876 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.708247 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.709526 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.710118 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.710357 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.711005 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:51.711008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:51.711008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:09:51.711284 (+ 276us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:51.711333 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:09:51.711345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:51.711428 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:51.711506 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:09:51.711586 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 00:09:51.712774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:09:51.725292 5903 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:51.725831 5903 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:51.742939 5903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d4251900da4946a821ef4cfdac910b: Generated new TSK 0
I20260924 00:09:51.743186 5903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:09:52.198915 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:51.752584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.98.62:35897)
0924 00:09:51.752654 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:51.752656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:51.752779 (+ 123us) server_negotiation.cc:407] Connection header received
0924 00:09:51.752864 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:51.752865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:51.752880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:51.752945 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:09:51.753130 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.753694 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.754848 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.755069 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.755124 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.755374 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:09:52.198662 (+443288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:09:52.198849 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 00:09:52.199024 5896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 00:09:52.199106 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:51.752529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35897 (local address 127.0.0.1:42746)
0924 00:09:51.752702 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:51.752714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:51.752784 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:51.752968 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:51.752968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:51.752970 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:09:51.753034 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.753035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.753785 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.753786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.754749 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:51.754751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:51.755155 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:51.755161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:51.755261 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:51.755268 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:09:52.198843 (+443575us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:09:52.198979 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260924 00:09:52.199534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5825
2026-09-24T00:09:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (644 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:52 dist-test-slave-3q8k krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:52 dist-test-slave-3q8k krb5kdc[5933](info): set up 2 sockets
Sep 24 00:09:52 dist-test-slave-3q8k krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260924 00:09:53.435627 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:53 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:09:53.559600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:53.642098 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:53.642360 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:53.642398 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:53.643566 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:53.643641 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:53.643666 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:53.643688 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:53.643714 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:53.645221 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:53.645498 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:53.645728 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:53.650717 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:53.650753 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:53.650878 5946 server_base.cc:1034] running on GCE node
W20260924 00:09:53.650729 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:53.651150 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:53.651506 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:53.652683 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790208593652672 us; error 46 us; skew 500 ppm
Sep 24 00:09:53 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:53.655117 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:09:53.655602 5946 webserver.cc:460] Webserver started at http://127.0.98.62:39803/ using document root <none> and password file <none>
I20260924 00:09:53.655853 5946 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:53.655930 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:53.656090 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:53.657256 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "1c998ecd844d498f8ca5c685b0ee5c8c"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "a69150f159c19b43598bde655ef589f6"
server_key_iv: "5be6a038804ed9f1dc4120e0f8d5b134"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.657676 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c998ecd844d498f8ca5c685b0ee5c8c"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "a69150f159c19b43598bde655ef589f6"
server_key_iv: "5be6a038804ed9f1dc4120e0f8d5b134"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.659104 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:53.659974 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:53.660189 5946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:53.660300 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "1c998ecd844d498f8ca5c685b0ee5c8c"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "a69150f159c19b43598bde655ef589f6"
server_key_iv: "5be6a038804ed9f1dc4120e0f8d5b134"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.660387 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:53.680972 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:53.683004 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:53.683353 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:53.687325 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43761
I20260924 00:09:53.687367 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43761 every 8 connection(s)
I20260924 00:09:53.687897 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:53.688608 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:53.690857 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c: Bootstrap starting.
I20260924 00:09:53.691425 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:53.691664 6014 log.cc:824] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:53.692293 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c: No bootstrap required, opened a new log
I20260924 00:09:53.693792 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260924 00:09:53.693928 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:53.693949 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c998ecd844d498f8ca5c685b0ee5c8c, State: Initialized, Role: FOLLOWER
I20260924 00:09:53.694096 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260924 00:09:53.694188 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:53.694212 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:53.694240 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:53.694801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 5946
I20260924 00:09:53.694952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:53.695132 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260924 00:09:53.695158 392 external_mini_cluster.cc:1428] Setting key 8cbb7adb73ebb16973a1f44f74dfa3dc
I20260924 00:09:53.695291 6014 leader_election.cc:302] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c998ecd844d498f8ca5c685b0ee5c8c; no voters:
I20260924 00:09:53.695477 6014 leader_election.cc:288] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:53.695564 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:53.695724 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:53.695779 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [term 1 LEADER]: Becoming Leader. State: Replica: 1c998ecd844d498f8ca5c685b0ee5c8c, State: Running, Role: LEADER
I20260924 00:09:53.695943 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260924 00:09:53.696416 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c998ecd844d498f8ca5c685b0ee5c8c. Latest consensus state: current_term: 1 leader_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } } }
I20260924 00:09:53.696420 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998ecd844d498f8ca5c685b0ee5c8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } } }
I20260924 00:09:53.696537 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:53.696547 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:53.696741 6028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:53.697253 6028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:53.699342 6028 catalog_manager.cc:1347] Generated new cluster ID: 4a89684718374f458965ac46fde51bad
I20260924 00:09:53.699398 6028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:53 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:53.706900 6028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:53.707640 6028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:53.708045 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:53.695856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.98.62:43761)
0924 00:09:53.696131 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:53.696140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:53.696174 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:09:53.696310 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:53.696318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:53.696390 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:53.697352 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:53.697478 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.698872 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.699929 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.700584 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.700709 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:53.703591 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:53.703595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:53.703595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:53.703608 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:53.706268 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:53.706710 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:53.706714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:53.706714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:53.706828 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:53.707093 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:53.707094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:53.707095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:53.707401 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:53.707593 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:53.707664 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:09:53.707822 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260924 00:09:53.716637 6028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c998ecd844d498f8ca5c685b0ee5c8c: Generated new TSK 0
I20260924 00:09:53.716861 6028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:09:53.749889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:53.833892 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:53.834111 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:53.834137 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:53.835566 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:53.835615 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:53.835644 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:53.837123 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:53.837320 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:53.837505 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:53.842633 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:53.842643 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:53.842657 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:53.842846 6040 server_base.cc:1034] running on GCE node
I20260924 00:09:53.843112 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:53.843393 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:53.844575 6040 hybrid_clock.cc:648] HybridClock initialized: now 1790208593844553 us; error 36 us; skew 500 ppm
Sep 24 00:09:53 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:53.847370 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:09:53.847836 6040 webserver.cc:460] Webserver started at http://127.0.98.1:35649/ using document root <none> and password file <none>
I20260924 00:09:53.848076 6040 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:53.848157 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:53.848330 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:53.849426 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "88af9184d93f40b887f54d1bd7df212d"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "27fd0d995067740bf402178dfefe069b"
server_key_iv: "daba4d926a56c88490ca0829a4370076"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.849812 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88af9184d93f40b887f54d1bd7df212d"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "27fd0d995067740bf402178dfefe069b"
server_key_iv: "daba4d926a56c88490ca0829a4370076"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.851378 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:53.852221 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:53.852427 6040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:53.852540 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "88af9184d93f40b887f54d1bd7df212d"
format_stamp: "Formatted at 2026-09-24 00:09:53 on dist-test-slave-3q8k"
server_key: "27fd0d995067740bf402178dfefe069b"
server_key_iv: "daba4d926a56c88490ca0829a4370076"
server_key_version: "encryptionkey@0"
I20260924 00:09:53.852622 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:53.868870 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:53.871476 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:53.871650 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:53.871896 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:53.872216 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:53.872274 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:53.872325 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:53.872352 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:53.878327 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40549
I20260924 00:09:53.878371 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40549 every 8 connection(s)
I20260924 00:09:53.878739 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:53.885087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6040
Sep 24 00:09:53 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:53.885180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:53.885380 392 external_mini_cluster.cc:1428] Setting key 0dd727b37a4d5e21de283da7d4d42cb1
I20260924 00:09:53.888919 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:53.879660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54663 (local address 127.0.98.62:43761)
0924 00:09:53.879731 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:53.879733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:53.879912 (+ 179us) server_negotiation.cc:407] Connection header received
0924 00:09:53.880868 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:53.880869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:53.880878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:53.880943 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:53.881603 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.882105 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.883129 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.883430 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.883493 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:53.885802 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:53.885804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:53.885804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:53.885813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:53.887423 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:53.887878 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:53.887879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:53.887880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:53.887941 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:53.888234 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:53.888235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:53.888235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:53.888358 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:53.888437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:53.888733 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 00:09:53.888818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:09:53.889147 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:53.879605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43761 (local address 127.0.98.1:54663)
0924 00:09:53.879775 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:53.879819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:53.880760 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:53.881024 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:53.881028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:53.881299 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:53.881516 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:53.881520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.882190 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.882191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:53.883020 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:53.883022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:53.883524 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:53.883524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:53.883642 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:53.884130 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:53.884139 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:53.885680 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:53.887529 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:53.887537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:53.887540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:53.887782 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:53.888013 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:53.888013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:53.888014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:53.888149 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:53.888430 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:53.888436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:53.888591 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 00:09:53.888713 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 00:09:53.888890 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260924 00:09:53.889851 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43761
I20260924 00:09:53.889971 6169 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:53.890166 6169 heartbeater.cc:511] Master 127.0.98.62:43761 requested a full tablet report, sending...
I20260924 00:09:53.890751 5978 ts_manager.cc:194] Registered new tserver with Master: 88af9184d93f40b887f54d1bd7df212d (127.0.98.1:40549)
I20260924 00:09:53.891486 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54663
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:09:53.912685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:53.994702 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:53.994917 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:53.994940 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:53.996285 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:53.996331 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:53.996359 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:53.998064 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:53.998330 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:53.998600 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:54.003821 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:54.003829 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:54.003832 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:54.004041 6174 server_base.cc:1034] running on GCE node
I20260924 00:09:54.004292 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:54.004540 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:54.005698 6174 hybrid_clock.cc:648] HybridClock initialized: now 1790208594005672 us; error 42 us; skew 500 ppm
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.008440 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:09:54.008865 6174 webserver.cc:460] Webserver started at http://127.0.98.2:36467/ using document root <none> and password file <none>
I20260924 00:09:54.009079 6174 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:54.009122 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:54.009217 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:54.010210 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "e17ff8f3057f4ebe9cb88079fc1121df"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "9d50ed4b4cdeb91da2e21d939b8b405e"
server_key_iv: "e628f73050e030e57281fd1deaba2bbc"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.010507 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e17ff8f3057f4ebe9cb88079fc1121df"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "9d50ed4b4cdeb91da2e21d939b8b405e"
server_key_iv: "e628f73050e030e57281fd1deaba2bbc"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.011870 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:54.012660 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.012863 6174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:54.012939 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "e17ff8f3057f4ebe9cb88079fc1121df"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "9d50ed4b4cdeb91da2e21d939b8b405e"
server_key_iv: "e628f73050e030e57281fd1deaba2bbc"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.013017 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:54.020637 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:54.022993 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:54.023110 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:54.023356 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:54.023607 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:54.023635 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.023658 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:54.023669 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.029351 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36511
I20260924 00:09:54.029393 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36511 every 8 connection(s)
I20260924 00:09:54.029744 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:54.038034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6174
I20260924 00:09:54.038144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:54.038324 392 external_mini_cluster.cc:1428] Setting key b77ac76166f4933788c837b9b1a16a74
I20260924 00:09:54.040241 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.030518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43761 (local address 127.0.98.2:53445)
0924 00:09:54.030678 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:54.030719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:54.031729 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:54.031992 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:54.031995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:54.032249 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:54.032448 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:54.032451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.033174 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.033175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:54.034197 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:54.034199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.034550 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.034550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:54.034652 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.035400 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:54.035408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:54.036879 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:54.038650 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:54.038676 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:54.038679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:54.039004 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:54.039299 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:54.039300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:54.039300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:54.039445 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:54.039747 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:54.039752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:54.039871 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 00:09:54.039955 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:09:54.040076 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 00:09:54.040247 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.030528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53445 (local address 127.0.98.62:43761)
0924 00:09:54.030597 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.030600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.030813 (+ 213us) server_negotiation.cc:407] Connection header received
0924 00:09:54.031825 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.031826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.031840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.031906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.032594 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.033083 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.034286 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.034463 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.034524 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.036971 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:54.036972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:54.036972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:54.036979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:54.038546 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.039100 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.039101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.039102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.039191 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.039524 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.039525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.039525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.039677 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:54.039751 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:54.040005 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 00:09:54.040105 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:09:54.040946 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43761
I20260924 00:09:54.041085 6303 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:54.041270 6303 heartbeater.cc:511] Master 127.0.98.62:43761 requested a full tablet report, sending...
I20260924 00:09:54.041704 5978 ts_manager.cc:194] Registered new tserver with Master: e17ff8f3057f4ebe9cb88079fc1121df (127.0.98.2:36511)
I20260924 00:09:54.042105 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53445
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:09:54.066605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:54.148694 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:54.148913 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:54.148938 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:54.150156 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:54.150203 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:54.150230 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:54.151825 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:54.152084 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:54.152333 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:54.157537 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:54.157567 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:54.157548 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:54.157827 6308 server_base.cc:1034] running on GCE node
I20260924 00:09:54.158057 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:54.158295 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:54.159483 6308 hybrid_clock.cc:648] HybridClock initialized: now 1790208594159460 us; error 38 us; skew 500 ppm
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.162146 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:09:54.162644 6308 webserver.cc:460] Webserver started at http://127.0.98.3:43267/ using document root <none> and password file <none>
I20260924 00:09:54.162915 6308 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:54.162997 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:54.163177 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:54.164384 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "80cf40162d9a425bbf092938d1d3a693"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "94160f660cd40f7f4476f2bb9bbec7cd"
server_key_iv: "17ed09f59a9cd70e3b473b1403279d1e"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.164801 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80cf40162d9a425bbf092938d1d3a693"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "94160f660cd40f7f4476f2bb9bbec7cd"
server_key_iv: "17ed09f59a9cd70e3b473b1403279d1e"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.166177 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:54.167022 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.167388 6308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:54.167501 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "80cf40162d9a425bbf092938d1d3a693"
format_stamp: "Formatted at 2026-09-24 00:09:54 on dist-test-slave-3q8k"
server_key: "94160f660cd40f7f4476f2bb9bbec7cd"
server_key_iv: "17ed09f59a9cd70e3b473b1403279d1e"
server_key_version: "encryptionkey@0"
I20260924 00:09:54.167601 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:54.183684 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:54.185932 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:54.186113 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:54.186360 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:54.186693 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:54.186728 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.186750 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:54.186805 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:54.192641 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43741
I20260924 00:09:54.192734 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43741 every 8 connection(s)
I20260924 00:09:54.193058 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:09:54.201747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6308
I20260924 00:09:54.201853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:54.202065 392 external_mini_cluster.cc:1428] Setting key be3c254c26fe25556e5cd891b194ede7
I20260924 00:09:54.203354 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.193864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34107 (local address 127.0.98.62:43761)
0924 00:09:54.193930 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.193932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.194129 (+ 197us) server_negotiation.cc:407] Connection header received
0924 00:09:54.195063 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.195064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.195074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.195137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.195922 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.196463 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.197562 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.197824 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.197886 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.200195 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:54.200197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:54.200197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:54.200205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:54.201676 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.202179 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.202180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.202181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.202248 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.202612 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.202612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.202613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.202754 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:54.202821 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:54.203160 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 00:09:54.203281 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:09:54.203526 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.193826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43761 (local address 127.0.98.3:34107)
0924 00:09:54.193997 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:54.194041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:54.194966 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:54.195193 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:54.195197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:54.195628 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:54.195831 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:54.195836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.196643 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.196644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:54.197483 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:54.197485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.197919 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.197920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:54.198038 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.198549 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:54.198557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:54.200091 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:54.201761 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:54.201768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:54.201770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:54.202080 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:54.202347 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:54.202347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:54.202348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:54.202537 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:54.202826 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:54.202831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:54.203008 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 00:09:54.203120 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:09:54.203303 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260924 00:09:54.203999 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43761
I20260924 00:09:54.204111 6437 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:54.204291 6437 heartbeater.cc:511] Master 127.0.98.62:43761 requested a full tablet report, sending...
I20260924 00:09:54.204674 5978 ts_manager.cc:194] Registered new tserver with Master: 80cf40162d9a425bbf092938d1d3a693 (127.0.98.3:43741)
I20260924 00:09:54.205053 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34107
I20260924 00:09:54.213792 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/test-admin.keytab.
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.231698 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 00:09:54.233389 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:54.232621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.98.1:40549)
0924 00:09:54.232794 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.232799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.232810 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:54.232837 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.232839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.232853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.232926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.233297 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32768: BlockingRecv error: recv got EOF from 127.0.0.1:32768 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260924 00:09:54.233498 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.234556 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.235512 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:54.243623 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.236164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.0.98.1:40549)
0924 00:09:54.236218 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.236219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.236229 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:54.236250 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.236251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.236262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.236314 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.236715 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.237440 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.238217 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.238571 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.238632 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.240805 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:54.240814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:54.240814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:54.240820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:54.242486 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.242805 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.242807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.242807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.242859 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.243026 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.243026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.243027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.243153 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:54.243332 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:54.243388 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 00:09:54.243454 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 00:09:54.245375 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:54.244910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.0.98.1:40549)
0924 00:09:54.244963 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.244964 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.244973 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:54.245023 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.245024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.245032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.245079 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.245303 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32778: BlockingRecv error: recv got EOF from 127.0.0.1:32778 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 00:09:54.245477 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.246476 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.247375 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:54.254135 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.247911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.98.1:40549)
0924 00:09:54.247970 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.247972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.247980 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:54.248030 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.248031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.248039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.248082 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.248405 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.248929 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.249652 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.249842 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.249892 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.251975 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:54.251976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:54.251976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:54.251982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:54.253169 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.253473 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.253474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.253474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.253534 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.253697 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.253697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.253698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.253783 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:54.253855 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:54.253938 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 00:09:54.253994 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 00:09:54.255949 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:09:54.255239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.98.1:40549)
0924 00:09:54.255304 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.255306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.255355 (+ 49us) server_negotiation.cc:407] Connection header received
0924 00:09:54.255459 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.255460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.255470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.255525 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.255888 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32798: BlockingRecv error: recv got EOF from 127.0.0.1:32798 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 00:09:54.256016 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790208568194258-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.257088 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:54.257999 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:09:54.265013 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:54.258501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.0.98.1:40549)
0924 00:09:54.258535 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:54.258537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:54.258590 (+ 53us) server_negotiation.cc:407] Connection header received
0924 00:09:54.258639 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:54.258639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:54.258646 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:54.258693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:54.259037 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.259571 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.260218 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:54.260393 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:54.260448 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:54.262445 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:54.262446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:54.262447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:54.262453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:54.263993 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.264313 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.264314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.264314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.264368 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:54.264608 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:54.264608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:54.264609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:54.264697 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:54.264768 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:54.264882 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:09:54.264951 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:09:54.265756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6040
I20260924 00:09:54.270304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6174
I20260924 00:09:54.274538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6308
I20260924 00:09:54.279047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 5946
2026-09-24T00:09:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1898 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[6457](info): set up 2 sockets
Sep 24 00:09:54 dist-test-slave-3q8k krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260924 00:09:55.391414 392 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:09:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 00:09:55.515149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:55.597265 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:55.597507 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:55.597533 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:55.598867 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:55.598915 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:55.598932 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:55.598985 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:55.599017 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:55.600775 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:55.601068 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:55.601341 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:55.607663 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:55.607640 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:55.607651 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:55.607717 6470 server_base.cc:1034] running on GCE node
I20260924 00:09:55.608276 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:55.608618 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:55.609781 6470 hybrid_clock.cc:648] HybridClock initialized: now 1790208595609766 us; error 29 us; skew 500 ppm
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:55.612504 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 00:09:55.612967 6470 webserver.cc:460] Webserver started at http://127.0.98.62:33213/ using document root <none> and password file <none>
I20260924 00:09:55.613236 6470 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:55.613320 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:55.613476 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:55.614547 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "321d9634ec5d44fc831f5360f9290504"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "7c583800bbbd76f1a844677b972ef925"
server_key_iv: "3b4400034ea3d8a7a78770eb444144fc"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.614917 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "321d9634ec5d44fc831f5360f9290504"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "7c583800bbbd76f1a844677b972ef925"
server_key_iv: "3b4400034ea3d8a7a78770eb444144fc"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.616500 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:55.617336 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:55.617522 6470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:55.617628 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "321d9634ec5d44fc831f5360f9290504"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "7c583800bbbd76f1a844677b972ef925"
server_key_iv: "3b4400034ea3d8a7a78770eb444144fc"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.617715 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:55.629318 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:55.631438 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:55.631625 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:55.635552 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45797
I20260924 00:09:55.635586 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45797 every 8 connection(s)
I20260924 00:09:55.635955 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:09:55.636946 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:55.639161 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Bootstrap starting.
I20260924 00:09:55.639942 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:55.640259 6538 log.cc:824] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:55.640290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6470
I20260924 00:09:55.640429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:09:55.640630 392 external_mini_cluster.cc:1428] Setting key 5672122a91975cdb826e4d51bd04d30f
I20260924 00:09:55.641039 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: No bootstrap required, opened a new log
I20260924 00:09:55.643173 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } }
I20260924 00:09:55.643369 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:55.643426 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321d9634ec5d44fc831f5360f9290504, State: Initialized, Role: FOLLOWER
I20260924 00:09:55.643577 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } }
I20260924 00:09:55.643652 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:09:55.643682 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:09:55.643726 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:55.644404 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } }
I20260924 00:09:55.644508 6538 leader_election.cc:302] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 321d9634ec5d44fc831f5360f9290504; no voters:
I20260924 00:09:55.644658 6538 leader_election.cc:288] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:09:55.644740 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:55.644878 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:55.644878 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 LEADER]: Becoming Leader. State: Replica: 321d9634ec5d44fc831f5360f9290504, State: Running, Role: LEADER
I20260924 00:09:55.645802 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } }
I20260924 00:09:55.647178 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321d9634ec5d44fc831f5360f9290504. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } }
I20260924 00:09:55.647226 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } }
I20260924 00:09:55.647315 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:55.647322 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
W20260924 00:09:55.647439 6557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:09:55.647490 6557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:09:55.647553 6560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:55.647750 6560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:55.649142 6560 catalog_manager.cc:1347] Generated new cluster ID: 57ccc3e6d83141088e7a54365b4a6592
I20260924 00:09:55.649191 6560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:09:55.662309 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:55.641184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.0.98.62:45797)
0924 00:09:55.641371 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:55.641380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:55.641413 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:09:55.642354 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:55.642363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:55.642440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:55.642590 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:55.643153 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.644844 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.645686 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.646330 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.646866 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:55.658368 (+ 11502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:55.658375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:55.658375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:55.658391 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:55.660560 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:55.660973 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:55.660977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:55.660978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:55.661100 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:55.661318 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:55.661319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:55.661319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:55.661655 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:55.661813 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:55.661899 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:09:55.662050 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260924 00:09:55.675571 6560 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:09:55.676327 6560 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:55.690851 6560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Generated new TSK 0
I20260924 00:09:55.691080 6560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 00:09:55.719252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:55.801299 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:55.801522 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:55.801546 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:55.802718 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:55.802763 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:55.802790 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:55.804275 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:55.804536 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:55.804785 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:55.809952 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:55.810038 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:55.810135 6564 server_base.cc:1034] running on GCE node
W20260924 00:09:55.809957 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:55.810444 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:55.810712 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:55.811892 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790208595811887 us; error 39 us; skew 500 ppm
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:55.814414 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:09:55.814858 6564 webserver.cc:460] Webserver started at http://127.0.98.1:41157/ using document root <none> and password file <none>
I20260924 00:09:55.815111 6564 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:55.815188 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:55.815409 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:55.816548 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "25c93a21847b4defa92167f368274440"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "00f8233f683e6b7afd7d1aaf5340a1b5"
server_key_iv: "b181b05afaf440ab71570b2d354d57aa"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.816931 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "25c93a21847b4defa92167f368274440"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "00f8233f683e6b7afd7d1aaf5340a1b5"
server_key_iv: "b181b05afaf440ab71570b2d354d57aa"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.818272 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:55.819111 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:55.819303 6564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:55.819410 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "25c93a21847b4defa92167f368274440"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "00f8233f683e6b7afd7d1aaf5340a1b5"
server_key_iv: "b181b05afaf440ab71570b2d354d57aa"
server_key_version: "encryptionkey@0"
I20260924 00:09:55.819497 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:55.863632 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:55.866039 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:55.866233 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:55.866525 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:55.866881 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:55.866942 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:55.866979 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:55.867017 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:55.873383 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45075
I20260924 00:09:55.873458 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45075 every 8 connection(s)
I20260924 00:09:55.873819 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:09:55.875198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6564
I20260924 00:09:55.875394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:09:55.875609 392 external_mini_cluster.cc:1428] Setting key 2ad2091542144150d7573085796a8b9f
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:09:55.885128 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:55.874807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52009 (local address 127.0.98.62:45797)
0924 00:09:55.874880 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:55.874882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:55.875049 (+ 167us) server_negotiation.cc:407] Connection header received
0924 00:09:55.875776 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:55.875777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:55.875790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:55.875837 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:55.877014 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.877536 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.878595 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.878821 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.878918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:55.881771 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:55.881772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:55.881772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:55.881779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:55.883428 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:55.884026 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:55.884027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:55.884028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:55.884087 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:55.884467 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:55.884468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:55.884468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:55.884616 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:55.884676 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:55.884977 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 00:09:55.885056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:09:55.885339 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:55.874702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:52009)
0924 00:09:55.874876 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:55.874928 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:55.875659 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:55.875918 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:55.875922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:55.876697 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:55.876899 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:55.876904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.877649 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.877650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:55.878489 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:55.878492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:55.878924 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:55.878925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:55.879079 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:55.879800 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:55.879809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:55.881646 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:55.883523 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:55.883531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:55.883534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:55.883905 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:55.884176 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:55.884177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:55.884178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:55.884356 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:55.884706 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:55.884712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:55.884849 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 00:09:55.884945 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:09:55.885078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260924 00:09:55.885895 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:55.886011 6693 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:55.886220 6693 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
I20260924 00:09:55.886869 6502 ts_manager.cc:194] Registered new tserver with Master: 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 00:09:55.887719 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 00:09:55.902932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:55.986557 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:55.986769 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:55.986793 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:55.988070 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:55.988118 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:55.988145 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:55.989701 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:55.989962 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:55.990202 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:55.995364 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:55.995425 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:55.995489 6698 server_base.cc:1034] running on GCE node
W20260924 00:09:55.995482 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:55.995848 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:55.996062 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:55.997210 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790208595997200 us; error 34 us; skew 500 ppm
Sep 24 00:09:55 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:55.999938 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:09:56.000411 6698 webserver.cc:460] Webserver started at http://127.0.98.2:41523/ using document root <none> and password file <none>
I20260924 00:09:56.000653 6698 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.000727 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.000891 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:56.001946 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "3230a4ecd5214bf7b0cd983f73d35dbf"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "2d295b7938f3ca01df639fbc72e66c52"
server_key_iv: "e2b0aa74829351ac7bd8985477af6b91"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.002316 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3230a4ecd5214bf7b0cd983f73d35dbf"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "2d295b7938f3ca01df639fbc72e66c52"
server_key_iv: "e2b0aa74829351ac7bd8985477af6b91"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.003759 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:09:56.004544 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.004722 6698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:56.004848 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "3230a4ecd5214bf7b0cd983f73d35dbf"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "2d295b7938f3ca01df639fbc72e66c52"
server_key_iv: "e2b0aa74829351ac7bd8985477af6b91"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.004930 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.032753 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.035413 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:56.035645 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.035943 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:56.036285 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:56.036347 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.036382 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:56.036412 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.042385 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34495
I20260924 00:09:56.042469 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34495 every 8 connection(s)
I20260924 00:09:56.042810 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:56.048209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6698
I20260924 00:09:56.048408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:09:56.048621 392 external_mini_cluster.cc:1428] Setting key 0703715312d9e02bf549b59658cc4678
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:09:56.053100 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.043569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34179 (local address 127.0.98.62:45797)
0924 00:09:56.043614 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.043616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.043822 (+ 206us) server_negotiation.cc:407] Connection header received
0924 00:09:56.044412 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.044413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.044426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.044501 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.045726 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.046364 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.047491 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.047674 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.047743 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.050083 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.050084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.050084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.050093 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.051685 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.052100 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.052101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.052102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.052165 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.052418 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.052419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.052419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.052565 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.052634 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.052900 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.052987 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:09:56.053128 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.043508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:34179)
0924 00:09:56.043684 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.043727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.044210 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.044611 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.044615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.045364 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.045632 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.045637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.046458 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.046459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.047410 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.047413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.047765 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.047765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.047876 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.048467 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.048478 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.049985 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.051778 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.051784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.051786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.052006 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.052240 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.052241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.052241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.052347 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.052634 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.052639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.052754 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.052838 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.052960 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260924 00:09:56.053797 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:56.053902 6827 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:56.054109 6827 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
I20260924 00:09:56.054468 6502 ts_manager.cc:194] Registered new tserver with Master: 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
I20260924 00:09:56.054864 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34179
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:09:56.074754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:56.156205 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:56.156406 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:56.156436 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:56.157512 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:56.157548 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:56.157574 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:56.158953 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:56.159142 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:56.159332 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:09:56.164681 6832 server_base.cc:1034] running on GCE node
W20260924 00:09:56.164635 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.164635 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.164644 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.165113 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.165380 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.166558 6832 hybrid_clock.cc:648] HybridClock initialized: now 1790208596166513 us; error 64 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.169255 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:09:56.169708 6832 webserver.cc:460] Webserver started at http://127.0.98.3:44887/ using document root <none> and password file <none>
I20260924 00:09:56.169974 6832 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.170048 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.170197 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:56.171336 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "7cc0121a1e579632b11a48ab6c9cfce2"
server_key_iv: "487474143003f9baa025db5f6678ad49"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.171692 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "7cc0121a1e579632b11a48ab6c9cfce2"
server_key_iv: "487474143003f9baa025db5f6678ad49"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.173038 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.173807 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.173982 6832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.174084 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "7cc0121a1e579632b11a48ab6c9cfce2"
server_key_iv: "487474143003f9baa025db5f6678ad49"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.174167 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.183758 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.186146 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:56.186321 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.186554 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:56.186863 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:56.186928 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.186971 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:56.187001 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.193035 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36029
I20260924 00:09:56.193074 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36029 every 8 connection(s)
I20260924 00:09:56.193423 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:56.199436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6832
I20260924 00:09:56.199503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:09:56.199707 392 external_mini_cluster.cc:1428] Setting key 56ea3830347dbc189b30628146b6d6c8
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:09:56.202986 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.194181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46263 (local address 127.0.98.62:45797)
0924 00:09:56.194270 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.194272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.194471 (+ 199us) server_negotiation.cc:407] Connection header received
0924 00:09:56.194991 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.194992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.195004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.195054 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.196071 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.196585 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.197605 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.197816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.197909 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.200265 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.200266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.200266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.200272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.201822 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.202180 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.202180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.202181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.202231 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.202461 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.202462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.202462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.202580 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.202631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.202866 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.202924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:09:56.203217 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.194153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:46263)
0924 00:09:56.194333 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.194377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.194879 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.195134 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.195137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.195751 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.195950 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.195955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.196701 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.196702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.197505 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.197506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.197919 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.197919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.198054 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.198563 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.198570 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.200184 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.201896 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.201901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.201903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.202110 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.202290 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.202290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.202291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.202384 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.202638 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.202642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.202757 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.202838 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.202960 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260924 00:09:56.203733 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:56.203812 6961 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:56.204018 6961 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
I20260924 00:09:56.204401 6502 ts_manager.cc:194] Registered new tserver with Master: 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029)
I20260924 00:09:56.204744 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46263
I20260924 00:09:56.211593 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 00:09:56.238727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38143
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45797,127.0.98.61:38143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:56.320389 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:56.320616 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:56.320639 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:56.321977 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:56.322053 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:56.322081 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:56.322096 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:56.322108 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:56.323697 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45797,127.0.98.61:38143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
W20260924 00:09:56.323798 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45797,127.0.98.61:38143'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 00:09:56.323911 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:56.324084 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.328836 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.328897 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.329131 6966 server_base.cc:1034] running on GCE node
W20260924 00:09:56.328963 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.329432 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.329738 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.330883 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790208596330853 us; error 49 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.333541 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 00:09:56.333945 6966 webserver.cc:460] Webserver started at http://127.0.98.61:41647/ using document root <none> and password file <none>
I20260924 00:09:56.334133 6966 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.334173 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.334261 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:56.335366 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/instance:
uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "6571c51cc270a92bd976e1e642b6951b"
server_key_iv: "71ffb3977a7ad155336ed6cef991f9b2"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.335680 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal/instance:
uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "6571c51cc270a92bd976e1e642b6951b"
server_key_iv: "71ffb3977a7ad155336ed6cef991f9b2"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.336994 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.337695 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.337852 6966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:56.337952 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "6571c51cc270a92bd976e1e642b6951b"
server_key_iv: "71ffb3977a7ad155336ed6cef991f9b2"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.338056 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.351781 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.353760 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:56.353874 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.357879 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38143
I20260924 00:09:56.357952 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38143 every 8 connection(s)
I20260924 00:09:56.358302 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:09:56.359342 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:56.360908 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:09:56.363458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 6966
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:09:56.371366 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.361701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.62:45797)
0924 00:09:56.361768 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.361770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.361976 (+ 206us) server_negotiation.cc:407] Connection header received
0924 00:09:56.362418 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.362419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.362429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.362492 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.363161 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.363871 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.365497 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.365701 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.365786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.368158 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.368160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.368160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.368169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.369762 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.370256 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.370258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.370258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.370320 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.370601 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.370602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.370602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.370759 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.370841 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.371141 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.371273 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:09:56.371572 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.361657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39822)
0924 00:09:56.361844 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.361892 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.362330 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.362582 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.362585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.362828 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.363038 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.363042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.363987 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.363988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.365396 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.365398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.365802 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.365803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.365930 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.366494 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.366503 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.368044 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.369861 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.369868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.369871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.370173 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.370394 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.370395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.370395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.370524 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.370844 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.370850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.370998 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.371118 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.371297 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":66,"thread_start_us":84,"threads_started":1}
I20260924 00:09:56.372228 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:09:56.373163 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.364014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.61:38143)
0924 00:09:56.364176 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.364181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.364201 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:09:56.364268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.364270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.364286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.364340 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.365096 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.365798 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.366732 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.367052 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.367139 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.369910 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.369918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.369919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.369933 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.371837 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.372213 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.372215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.372215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.372266 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.372489 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.372490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.372491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.372671 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.372840 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.372928 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.373056 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:09:56.379292 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.372509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60400)
0924 00:09:56.372565 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.372576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.372612 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.372891 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.372891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.373074 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.373177 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.373179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.373953 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.373954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.374414 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.374416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.374816 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.374817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.374866 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.375199 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.375203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.376380 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.378467 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.378468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.378468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.378621 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.378795 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.378795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.378795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.378842 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.379012 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.379013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.379045 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.379094 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.379165 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:09:56.379416 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.372582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.0.98.61:38143)
0924 00:09:56.372717 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.372719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.372723 (+ 4us) server_negotiation.cc:407] Connection header received
0924 00:09:56.372737 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.372738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.372744 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.372847 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.373265 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.373871 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.374509 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.374751 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.374805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.376522 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.376523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.376524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.376531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.378399 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.378689 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.378690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.378690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.378739 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.378879 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.378880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.378880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.378963 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.379005 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.379118 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.379277 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260924 00:09:56.380668 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Bootstrap starting.
I20260924 00:09:56.381258 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:56.381511 7034 log.cc:824] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:56.382217 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: No bootstrap required, opened a new log
I20260924 00:09:56.383620 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } }
I20260924 00:09:56.383735 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:56.383757 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ad2d49f51c4c7f87c65e4cde39a77a, State: Initialized, Role: FOLLOWER
I20260924 00:09:56.383877 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } }
I20260924 00:09:56.384132 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } } }
I20260924 00:09:56.384197 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:56.384213 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:09:56.385825 7052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:09:56.385905 7052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:09:56.387619 6966 master_runner.cc:428] Detected that this master 69ad2d49f51c4c7f87c65e4cde39a77a is joining an existing cluster
I20260924 00:09:56.387669 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38143
I20260924 00:09:56.395267 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.388800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.98.61:38143)
0924 00:09:56.388845 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.388847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.388853 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:09:56.388879 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.388879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.388887 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.388927 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.389285 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.389848 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.390741 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.391075 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.391152 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.392366 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.392367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.392367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.392374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.394045 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394538 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.394539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.394540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.394588 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394759 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.394760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.394760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.394865 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.394905 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.395070 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.395143 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:09:56.395422 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.388660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.98.62:45797)
0924 00:09:56.388705 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.388708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.388953 (+ 245us) server_negotiation.cc:407] Connection header received
0924 00:09:56.389058 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.389059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.389069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.389124 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.389504 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.390248 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.391086 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.391425 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.391521 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.392411 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.392412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.392412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.392419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.394033 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394546 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.394547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.394548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.394602 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394828 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.394829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.394829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.394954 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.395048 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.395209 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.395336 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:09:56.395433 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.388601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39826)
0924 00:09:56.388773 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.388785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.388969 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.389129 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.389130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.389351 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.389411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.389412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.390339 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.390339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.390999 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.391000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.391529 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.391530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.391613 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.391954 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.391958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.392316 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.394144 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.394146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.394450 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.394683 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.394684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.394754 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.395042 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.395043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.395097 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.395185 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.395328 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260924 00:09:56.396003 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.388645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60404)
0924 00:09:56.388773 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.388784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.388831 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.388936 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.388936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.389123 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.389200 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.389202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.389937 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.389938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.390652 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.390654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.391164 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.391165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.391249 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.391771 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.391776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.392277 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.394130 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.394131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.394460 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.394650 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.394650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.394650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.394698 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.394943 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.394944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.394976 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.395683 (+ 707us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.395804 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:09:56.405233 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.397999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.98.61:38143)
0924 00:09:56.398062 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.398064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.398188 (+ 124us) server_negotiation.cc:407] Connection header received
0924 00:09:56.398281 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.398282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.398292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.398346 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.398768 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.399379 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.400180 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.400382 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.400471 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.402563 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.402564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.402564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.402570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.404056 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.404530 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.404531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.404531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.404579 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.404778 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.404779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.404779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.404870 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.404918 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.405118 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.405173 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:09:56.405310 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.397933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60408)
0924 00:09:56.398067 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.398111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.398216 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.398352 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.398356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.398633 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.398678 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.398679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.399472 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.399473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.400070 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.400072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.400473 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.400474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.400576 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.401049 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.401054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.402471 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.404170 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.404176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.404178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.404445 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.404649 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.404650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.404650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.404717 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.404938 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.404944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.404999 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.405089 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.405205 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260924 00:09:56.405920 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38143
}
attrs {
promote: true
}
}
dest_uuid: "321d9634ec5d44fc831f5360f9290504"
cas_config_opid_index: -1
I20260924 00:09:56.406250 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } }
I20260924 00:09:56.406790 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } }
I20260924 00:09:56.407141 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:56.407925 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38143
I20260924 00:09:56.407955 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 00:09:56.408133 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } }
I20260924 00:09:56.408170 6966 master.cc:537] Master@127.0.98.61:38143 shutting down...
I20260924 00:09:56.408206 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:56.409194 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:09:56.410364 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 00:09:56.410795 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 -> Peer 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61:38143): Couldn't send request to peer 69ad2d49f51c4c7f87c65e4cde39a77a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:09:56.410863 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:56.410952 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:56.411000 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: stopping tablet replica
I20260924 00:09:56.423938 6966 master.cc:552] Master@127.0.98.61:38143 shutdown complete.
I20260924 00:09:56.424006 6966 master_runner.cc:305] Clearing existing system tablet
I20260924 00:09:56.424144 6966 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.424196 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.424957 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.425478 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.425633 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.425699 6966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.425719 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "6571c51cc270a92bd976e1e642b6951b"
server_key_iv: "71ffb3977a7ad155336ed6cef991f9b2"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.426268 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:56.428107 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:09:56.428165 6966 log.cc:1196] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 00:09:56.428373 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Deleting consensus metadata
I20260924 00:09:56.428503 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:45797
I20260924 00:09:56.429191 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45797
I20260924 00:09:56.436002 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.429537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39838)
0924 00:09:56.429690 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.429700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.429764 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.429900 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.429901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.430056 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.430120 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.430121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.431069 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.431070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.431690 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.431692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.432164 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.432165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.432213 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.432734 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.432738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.433035 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.434918 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.434918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.434919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.435118 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.435394 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.435394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.435395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.435457 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.435742 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.435743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.435781 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.435833 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.435912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260924 00:09:56.436084 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.429595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.98.62:45797)
0924 00:09:56.429638 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.429640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.429780 (+ 140us) server_negotiation.cc:407] Connection header received
0924 00:09:56.429815 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.429816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.429828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.429897 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.430189 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.430970 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.431777 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.432084 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.432157 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.433131 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.433133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.433133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.433141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.434818 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.435204 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.435205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.435206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.435318 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.435533 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.435534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.435535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.435675 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.435746 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.435900 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.435981 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:09:56.436719 6522 tablet_copy_service.cc:140] P 321d9634ec5d44fc831f5360f9290504: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 69ad2d49f51c4c7f87c65e4cde39a77a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39838)
I20260924 00:09:56.436834 6522 tablet_copy_service.cc:161] P 321d9634ec5d44fc831f5360f9290504: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 69ad2d49f51c4c7f87c65e4cde39a77a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39838: session id = 69ad2d49f51c4c7f87c65e4cde39a77a-00000000000000000000000000000000
I20260924 00:09:56.437477 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:09:56.438136 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:09:56.439060 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: tablet copy: Starting download of 0 data blocks...
I20260924 00:09:56.439296 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: tablet copy: Starting download of 1 WAL segments...
I20260924 00:09:56.440222 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:09:56.441092 6522 tablet_copy_service.cc:342] P 321d9634ec5d44fc831f5360f9290504: Request end of tablet copy session 69ad2d49f51c4c7f87c65e4cde39a77a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39838
I20260924 00:09:56.441152 6522 tablet_copy_service.cc:433] P 321d9634ec5d44fc831f5360f9290504: ending tablet copy session 69ad2d49f51c4c7f87c65e4cde39a77a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 69ad2d49f51c4c7f87c65e4cde39a77a
I20260924 00:09:56.443674 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.444998 7074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.445079 7073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.445194 6966 server_base.cc:1034] running on GCE node
W20260924 00:09:56.445243 7076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.445480 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.445621 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.446749 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790208596446742 us; error 25 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.448227 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 00:09:56.448524 6966 webserver.cc:460] Webserver started at http://127.0.98.61:33667/ using document root <none> and password file <none>
I20260924 00:09:56.448647 6966 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.448709 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.449333 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.449874 7083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.450042 6966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.450100 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "6571c51cc270a92bd976e1e642b6951b"
server_key_iv: "71ffb3977a7ad155336ed6cef991f9b2"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.450157 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.463176 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.464800 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.467801 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38143
I20260924 00:09:56.468122 7135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38143 every 8 connection(s)
I20260924 00:09:56.468233 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:09:56.468940 7136 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:09:56.469390 7136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Bootstrap starting.
I20260924 00:09:56.471769 7136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:09:56.472011 7136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: Bootstrap complete.
I20260924 00:09:56.472225 7136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } }
I20260924 00:09:56.472321 7136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69ad2d49f51c4c7f87c65e4cde39a77a, State: Initialized, Role: LEARNER
I20260924 00:09:56.472362 7136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } }
I20260924 00:09:56.472611 7136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:56.472580 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } }
I20260924 00:09:56.472672 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: LEARNER
I20260924 00:09:56.473999 7150 catalog_manager.cc:1259] Loaded cluster ID: 57ccc3e6d83141088e7a54365b4a6592
I20260924 00:09:56.474071 7150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: loading cluster ID for follower catalog manager: success
I20260924 00:09:56.475312 7150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: acquiring CA information for follower catalog manager: success
I20260924 00:09:56.475657 7150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:09:56.482216 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.474822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.61:38143)
0924 00:09:56.474912 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.474915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.474924 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:56.474950 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.474951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.474961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.475052 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.475792 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.476415 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.477470 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.477665 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.477717 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.479154 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.479156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.479156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.479162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.480985 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.481441 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.481441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.481442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.481491 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.481732 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.481733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.481733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.481833 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.481908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.482072 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.482130 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":40,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 00:09:56.508582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40151
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:09:56.592253 7154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:56.592502 7154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:56.592540 7154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:56.593659 7154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:09:56.593741 7154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:56.593768 7154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:56.593798 7154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:09:56.593820 7154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:09:56.595410 7154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:56.595698 7154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:56.595938 7154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.601177 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.601136 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.601208 7154 server_base.cc:1034] running on GCE node
W20260924 00:09:56.601154 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.601590 7154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.601923 7154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.603096 7154 hybrid_clock.cc:648] HybridClock initialized: now 1790208596603092 us; error 41 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.605664 7154 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 00:09:56.606227 7154 webserver.cc:460] Webserver started at http://127.0.98.60:41267/ using document root <none> and password file <none>
I20260924 00:09:56.606560 7154 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.606642 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.606777 7154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:09:56.608327 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/instance:
uuid: "fc409f33c33c4e2a84d324fc033beb19"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "cbde0c30dbf6f7b5aaf043deab43087c"
server_key_iv: "b4fa6af2e83a9ecd330396206ee4337f"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.608774 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal/instance:
uuid: "fc409f33c33c4e2a84d324fc033beb19"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "cbde0c30dbf6f7b5aaf043deab43087c"
server_key_iv: "b4fa6af2e83a9ecd330396206ee4337f"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.610388 7154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:09:56.611160 7169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.611340 7154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:56.611445 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "fc409f33c33c4e2a84d324fc033beb19"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "cbde0c30dbf6f7b5aaf043deab43087c"
server_key_iv: "b4fa6af2e83a9ecd330396206ee4337f"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.611517 7154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.652029 7154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.654088 7154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:56.654284 7154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.658326 7154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40151
I20260924 00:09:56.658452 7221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40151 every 8 connection(s)
I20260924 00:09:56.658735 7154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:09:56.659819 7222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:56.661386 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:09:56.664206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7154
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:56.670904 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.662239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.0.98.62:45797)
0924 00:09:56.662303 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.662311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.662464 (+ 153us) server_negotiation.cc:407] Connection header received
0924 00:09:56.662931 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.662932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.662943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.663009 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.663756 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.664309 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.665517 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.665709 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.665793 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.668009 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.668011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.668011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.668021 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.669474 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.669906 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.669907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.669907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.669971 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.670267 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.670267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.670268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.670369 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.670459 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.670730 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.670810 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:09:56.671092 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.662117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39842)
0924 00:09:56.662302 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.662346 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.662832 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.663091 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.663094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.663448 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.663658 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.663663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.664415 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.664416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.665391 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.665393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.665799 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.665800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.665924 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.666474 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.666482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.667915 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.669565 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.669570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.669572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.669786 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.670047 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.670048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.670048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.670193 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.670453 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.670460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.670594 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.670710 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.670858 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260924 00:09:56.671800 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:09:56.673835 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.664737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.60:40151)
0924 00:09:56.664930 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.664934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.664951 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:09:56.664969 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.664971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.664986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.665077 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.665673 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.666373 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.667111 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.667479 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.667572 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.670381 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.670387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.670388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.670394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.672567 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.672963 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.672966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.672966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.673023 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.673231 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.673232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.673233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.673371 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.673576 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.673657 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.673730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:09:56.678655 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.672087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60438)
0924 00:09:56.672134 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.672144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.672183 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.672348 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.672349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.672590 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.672646 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.672648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.673365 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.673365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.674180 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.674182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.674623 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.674624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.674682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.674998 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.675001 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.676218 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.677629 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.677630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.677630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.677803 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.678033 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.678034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.678034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.678090 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.678351 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.678351 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.678392 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.678462 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.678564 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:09:56.678663 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.672171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.0.98.61:38143)
0924 00:09:56.672219 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.672221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.672230 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:56.672255 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.672257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.672271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.672339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.672738 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.673266 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.674265 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.674535 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.674615 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.676311 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.676312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.676313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.676320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.677520 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.677896 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.677897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.677897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.677953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.678168 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.678169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.678170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.678282 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.678358 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.678497 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.678576 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:09:56.679188 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:56.685650 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.679507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36466)
0924 00:09:56.679561 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.679572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.679611 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.679717 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.679718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.679819 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.679845 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.679845 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.680446 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.680447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.681021 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.681022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.681365 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.681366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.681414 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.681901 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.681906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.683165 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.684845 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.684846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.684846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.685000 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.685170 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.685170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.685170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.685217 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.685410 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.685411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.685441 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.685490 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.685567 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:09:56.685698 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.679575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.0.98.60:40151)
0924 00:09:56.679604 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.679606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.679612 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:09:56.679663 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.679663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.679672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.679715 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.679891 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.680330 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.681103 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.681282 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.681331 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.683301 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.683302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.683303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.683311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.684774 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.685068 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.685069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.685069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.685113 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.685272 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.685273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.685273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.685359 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.685405 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.685507 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.685607 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":14}
I20260924 00:09:56.687032 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Bootstrap starting.
I20260924 00:09:56.687973 7222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:56.688272 7222 log.cc:824] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:56.688982 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: No bootstrap required, opened a new log
I20260924 00:09:56.690357 7222 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } }
I20260924 00:09:56.690532 7222 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:56.690584 7222 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc409f33c33c4e2a84d324fc033beb19, State: Initialized, Role: FOLLOWER
I20260924 00:09:56.690742 7222 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } }
I20260924 00:09:56.691071 7222 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:56.691035 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } } }
I20260924 00:09:56.691129 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:09:56.692777 7238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:09:56.692864 7238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:09:56.695317 7154 master_runner.cc:428] Detected that this master fc409f33c33c4e2a84d324fc033beb19 is joining an existing cluster
I20260924 00:09:56.695369 7154 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40151
I20260924 00:09:56.704159 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.98.61:38143)
0924 00:09:56.696900 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.696901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.696909 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:56.696931 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.696932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.696942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.696996 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.697871 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698617 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699711 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.700022 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.700066 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.701562 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.701563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.701563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.701569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.702881 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.703323 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.703324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.703325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.703384 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.703627 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.703628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.703628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.703747 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.703828 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.703981 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.704060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:09:56.704173 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60450)
0924 00:09:56.696609 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.696620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.696667 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.697002 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.697003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.697246 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.697768 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.697770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698715 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.698716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.699614 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.699616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.700144 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.700145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.700541 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.701029 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.701034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.701459 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.702979 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.702980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.702981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.703260 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.703470 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.703471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.703471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.703552 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.703821 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.703822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.703865 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.703956 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.704062 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260924 00:09:56.725212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6564
I20260924 00:09:56.726696 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.0.98.60:40151)
0924 00:09:56.696940 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.696942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.697115 (+ 173us) server_negotiation.cc:407] Connection header received
0924 00:09:56.697195 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.697196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.697204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.697229 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.697623 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698328 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699509 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.699806 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699845 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.723720 (+ 23875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.723721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.723722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.723731 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.725461 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.725853 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.725854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.725855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.725903 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.726086 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.726086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.726087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.726228 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.726267 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.726450 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.726547 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:09:56.726990 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36468)
0924 00:09:56.697065 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.697074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.697135 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.697280 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.697281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.697495 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.697551 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.697553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698427 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.698428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.699420 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.699421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.699937 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.700000 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.700489 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.700494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.723608 (+ 23114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.725558 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.725559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.725559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.725772 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.725966 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.725966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.725967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.726026 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.726305 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.726306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.726352 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.726772 (+ 420us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.726874 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":39,"thread_start_us":72,"threads_started":1}
I20260924 00:09:56.729153 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.98.62:45797)
0924 00:09:56.696566 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.696569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.696886 (+ 317us) server_negotiation.cc:407] Connection header received
0924 00:09:56.697013 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.697014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.697041 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.697075 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.697605 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698348 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699514 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.699806 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699845 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.724559 (+ 24714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.724561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.724562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.724574 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.727880 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.728280 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.728281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.728281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.728336 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.728539 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.728540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.728540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.728687 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.728729 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.728925 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.729009 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:09:56.729486 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.696462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39854)
0924 00:09:56.696806 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.696815 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.696924 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.697142 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.697143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.697445 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.697509 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.697511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.698442 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.698443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.699415 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.699419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.699936 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.699937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.700000 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.723965 (+ 23965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.723969 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.724444 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.727966 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.727967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.727967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.728198 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.728403 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.728404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.728404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.728459 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.728772 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.728773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.728817 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.729244 (+ 427us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.729355 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":43,"thread_start_us":55,"threads_started":1}
I20260924 00:09:56.731448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45075
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45797
--webserver_port=41157
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:56.741845 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.733438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.98.60:40151)
0924 00:09:56.733478 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.733480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.733489 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:56.733545 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.733546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.733556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.733620 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.733896 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.734624 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.735909 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.736209 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.736290 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.738700 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.738701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.738702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.738710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.740517 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.740928 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.740930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.740930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.740992 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.741220 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.741221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.741222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.741386 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.741463 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.741638 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.741717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:09:56.742028 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.733364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36476)
0924 00:09:56.733422 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.733436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.733477 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.733644 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.733645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.733779 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.733821 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.733822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.734724 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.734725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.735776 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.735779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.736309 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.736310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.736466 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.736970 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.736976 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.738600 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.740612 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.740613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.740614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.740831 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.741075 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.741076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.741077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.741144 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.741467 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.741468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.741515 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.741590 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.741927 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:09:56.742898 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40151: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40151
}
attrs {
promote: true
}
}
dest_uuid: "321d9634ec5d44fc831f5360f9290504"
cas_config_opid_index: 5
I20260924 00:09:56.743148 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } }
I20260924 00:09:56.743516 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.744410 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40151
I20260924 00:09:56.744493 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.744554 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:56.745090 7196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:56.746497 7196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 00:09:56.746731 7154 master.cc:537] Master@127.0.98.60:40151 shutting down...
I20260924 00:09:56.746759 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:09:56.747651 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 -> Peer fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60:40151): Couldn't send request to peer fc409f33c33c4e2a84d324fc033beb19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:09:56.750083 7154 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:56.750172 7154 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:56.750214 7154 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: stopping tablet replica
I20260924 00:09:56.751132 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.744873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.98.61:38143)
0924 00:09:56.744918 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.744920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.746950 (+ 2030us) server_negotiation.cc:407] Connection header received
0924 00:09:56.747078 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.747080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.747102 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.747142 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:56.747464 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.748237 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.749798 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.750471 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.750547 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.750659 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.751022 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.751064 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:09:56.751605 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.744787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60462)
0924 00:09:56.746888 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.746900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.746982 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.747330 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.747331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.747332 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:56.747375 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.747376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.748336 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.748337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.749679 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.749682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.750793 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.750794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.750908 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.750930 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.751382 (+ 452us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.751427 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2082}
I20260924 00:09:56.751955 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 LEARNER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 00:09:56.752148 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:09:56.752660 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.753679 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321d9634ec5d44fc831f5360f9290504. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.753750 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: LEARNER
I20260924 00:09:56.753779 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: true } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.753815 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: LEARNER
I20260924 00:09:56.754061 7062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: attempting to promote NON_VOTER 69ad2d49f51c4c7f87c65e4cde39a77a to VOTER
I20260924 00:09:56.754261 7062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } }
I20260924 00:09:56.754776 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.754646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36478)
0924 00:09:56.754704 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.754746 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:09:56.754802 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 LEARNER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 00:09:56.754976 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:09:56.755208 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.755080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36488)
0924 00:09:56.755163 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.755186 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 00:09:56.755416 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 -> Peer fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60:40151): Couldn't send request to peer fc409f33c33c4e2a84d324fc033beb19. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:09:56.755579 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.756474 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.756551 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:56.756596 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.756644 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:56.756903 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.757733 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.757867 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:09:56.763456 7154 master.cc:552] Master@127.0.98.60:40151 shutdown complete.
I20260924 00:09:56.763515 7154 master_runner.cc:305] Clearing existing system tablet
I20260924 00:09:56.763649 7154 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.763697 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.764459 7154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.765038 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.765190 7154 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.765239 7154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.765261 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "fc409f33c33c4e2a84d324fc033beb19"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "cbde0c30dbf6f7b5aaf043deab43087c"
server_key_iv: "b4fa6af2e83a9ecd330396206ee4337f"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.765967 7154 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:56.767681 7154 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:09:56.767751 7154 log.cc:1196] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 00:09:56.767941 7154 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Deleting consensus metadata
I20260924 00:09:56.768092 7154 master_runner.cc:315] Copying system tablet from 127.0.98.62:45797
I20260924 00:09:56.768713 7154 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45797
I20260924 00:09:56.776191 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.769099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.0.98.62:45797)
0924 00:09:56.769144 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.769147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.769203 (+ 56us) server_negotiation.cc:407] Connection header received
0924 00:09:56.769271 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.769272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.769282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.769358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.769735 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.770489 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.771572 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.771853 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.771892 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.773176 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.773178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.773178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.773185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.774845 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.775341 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.775342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.775342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.775401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.775599 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.775600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.775600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.775757 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.775801 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.776012 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.776089 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:09:56.776508 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.769027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:39866)
0924 00:09:56.769145 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.769156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.769216 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.769389 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.769390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.769586 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.769644 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.769646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.770591 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.770592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.771474 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.771477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.771968 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.771969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.772049 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.772600 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.772605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.773027 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.774938 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.774939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.774940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.775181 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.775476 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.775476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.775477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.775532 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.775848 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.775849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.775895 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.776317 (+ 422us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.776420 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260924 00:09:56.776814 6522 tablet_copy_service.cc:140] P 321d9634ec5d44fc831f5360f9290504: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc409f33c33c4e2a84d324fc033beb19 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39866)
I20260924 00:09:56.776907 6522 tablet_copy_service.cc:161] P 321d9634ec5d44fc831f5360f9290504: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc409f33c33c4e2a84d324fc033beb19 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39866: session id = fc409f33c33c4e2a84d324fc033beb19-00000000000000000000000000000000
I20260924 00:09:56.777355 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:09:56.777894 7154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:09:56.779136 7154 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: tablet copy: Starting download of 0 data blocks...
I20260924 00:09:56.779295 7154 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: tablet copy: Starting download of 1 WAL segments...
I20260924 00:09:56.780150 7154 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:09:56.781067 6522 tablet_copy_service.cc:342] P 321d9634ec5d44fc831f5360f9290504: Request end of tablet copy session fc409f33c33c4e2a84d324fc033beb19-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39866
I20260924 00:09:56.781131 6522 tablet_copy_service.cc:433] P 321d9634ec5d44fc831f5360f9290504: ending tablet copy session fc409f33c33c4e2a84d324fc033beb19-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc409f33c33c4e2a84d324fc033beb19
I20260924 00:09:56.783759 7154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.784961 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.785208 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.787395 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.791406 7154 server_base.cc:1034] running on GCE node
I20260924 00:09:56.791551 7154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.791699 7154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.792829 7154 hybrid_clock.cc:648] HybridClock initialized: now 1790208596792801 us; error 47 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.794390 7154 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 00:09:56.794673 7154 webserver.cc:460] Webserver started at http://127.0.98.60:38919/ using document root <none> and password file <none>
I20260924 00:09:56.794816 7154 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.794868 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.795735 7154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:56.796435 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.796612 7154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:56.796671 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "fc409f33c33c4e2a84d324fc033beb19"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "cbde0c30dbf6f7b5aaf043deab43087c"
server_key_iv: "b4fa6af2e83a9ecd330396206ee4337f"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.796730 7154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.820719 7154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.823020 7154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.826218 7154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40151
I20260924 00:09:56.826311 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40151 every 8 connection(s)
I20260924 00:09:56.826553 7154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:09:56.827555 7322 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:09:56.828068 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Bootstrap starting.
I20260924 00:09:56.831139 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:09:56.831390 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Bootstrap complete.
I20260924 00:09:56.831619 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } }
I20260924 00:09:56.831717 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc409f33c33c4e2a84d324fc033beb19, State: Initialized, Role: LEARNER
I20260924 00:09:56.831759 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } }
I20260924 00:09:56.831940 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:09:56.833431 7339 catalog_manager.cc:1259] Loaded cluster ID: 57ccc3e6d83141088e7a54365b4a6592
I20260924 00:09:56.833530 7339 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: loading cluster ID for follower catalog manager: success
I20260924 00:09:56.834738 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:56.834834 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:09:56.835265 7339 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: acquiring CA information for follower catalog manager: success
W20260924 00:09:56.835143 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:56.835450 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:56.835508 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 00:09:56.835574 7339 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 00:09:56.836727 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:56.836771 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:56.836798 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:09:56.838325 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41157
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:56.838581 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:56.838840 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.843850 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.843856 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.843892 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.844342 7247 server_base.cc:1034] running on GCE node
I20260924 00:09:56.844545 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.844836 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.845996 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790208596845966 us; error 43 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.848889 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:09:56.849355 7247 webserver.cc:460] Webserver started at http://127.0.98.1:41157/ using document root <none> and password file <none>
I20260924 00:09:56.849596 7247 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.849682 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.851106 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.851774 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.851938 7247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:09:56.852056 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "25c93a21847b4defa92167f368274440"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "00f8233f683e6b7afd7d1aaf5340a1b5"
server_key_iv: "b181b05afaf440ab71570b2d354d57aa"
server_key_version: "encryptionkey@0"
I20260924 00:09:56.852411 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:56.869449 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:56.872118 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:56.872287 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:56.872555 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:56.872928 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:56.872999 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.873037 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:56.873061 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.880035 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45075
I20260924 00:09:56.880079 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45075 every 8 connection(s)
I20260924 00:09:56.880455 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:56.889127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7247
I20260924 00:09:56.889281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6698
I20260924 00:09:56.891171 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43617 (local address 127.0.98.62:45797)
0924 00:09:56.881510 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.881513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.881524 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:56.882091 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.882092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.882105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.882161 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.882961 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.883585 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.884631 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.884893 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.884953 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.887641 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.887642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.887642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.887649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.889417 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.889924 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.889926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.889926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.889990 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.890307 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.890308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.890309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.890476 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.890555 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.890924 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.891024 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:09:56.891357 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:43617)
0924 00:09:56.881361 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.881403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.881985 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.882261 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.882267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.882672 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.882863 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.882867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.883696 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.883697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.884517 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.884519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.885026 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.885027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.885181 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.885898 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.885903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.887539 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.889527 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.889535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.889537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.889862 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.890075 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.890076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.890076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.890219 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.890566 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.890573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.890754 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.890880 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.891056 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":104,"threads_started":1}
I20260924 00:09:56.891882 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:56.891980 7466 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:56.892220 7466 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
I20260924 00:09:56.892320 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59051 (local address 127.0.98.60:40151)
0924 00:09:56.881333 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.881336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.881691 (+ 355us) server_negotiation.cc:407] Connection header received
0924 00:09:56.882127 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.882129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.882145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.882218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.882983 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.884009 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.884925 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.885158 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.885252 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.888720 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.888722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.888722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.888729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.890715 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.891087 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.891088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.891089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.891146 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.891724 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.891725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.891725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.891867 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.891938 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.892139 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.892231 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260924 00:09:56.892396 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.98.1:59051)
0924 00:09:56.881622 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.881634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.882006 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.882271 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.882272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.882743 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.882864 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.882870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.884094 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.884095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.884867 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.884868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.885308 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.885309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.885368 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.886051 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.886056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.888608 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.890802 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.890803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.890804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.891008 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.891342 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.891343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.891343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.891639 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.891957 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.891959 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.892007 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.892089 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.892200 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260924 00:09:56.892689 6502 ts_manager.cc:194] Re-registered known tserver with Master: 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
I20260924 00:09:56.893245 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43617
I20260924 00:09:56.895295 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40151
I20260924 00:09:56.895358 7465 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:56.895430 7465 heartbeater.cc:511] Master 127.0.98.60:40151 requested a full tablet report, sending...
I20260924 00:09:56.895856 7286 ts_manager.cc:194] Registered new tserver with Master: 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
I20260924 00:09:56.896538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34495
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45797
--webserver_port=41523
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:09:56.898166 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60993 (local address 127.0.98.61:38143)
0924 00:09:56.881711 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:56.881713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:56.882261 (+ 548us) server_negotiation.cc:407] Connection header received
0924 00:09:56.882296 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:56.882298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:56.882310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:56.882360 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:56.882972 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.883787 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.884847 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.885128 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.885194 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.888116 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:56.888118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:56.888118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:56.888126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:56.896737 (+ 8611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.897221 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.897223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.897223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.897288 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:56.897542 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:56.897543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:56.897544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:56.897702 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:56.897773 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:56.897988 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 00:09:56.898070 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:56.898185 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:56.881509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.98.1:60993)
0924 00:09:56.882188 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:56.882198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:56.882246 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:56.882371 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:56.882372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:56.882698 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:56.882864 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.882870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.883880 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.883880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.884760 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:56.884761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:56.885217 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:56.885218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:56.885288 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:56.885817 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:56.885825 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:56.888001 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:56.896869 (+ 8868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.896870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.896871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.897132 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.897386 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:56.897387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:56.897388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:56.897455 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:56.897806 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:56.897808 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:56.897868 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 00:09:56.897959 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 00:09:56.898071 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":64,"threads_started":1}
I20260924 00:09:56.898743 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38143
I20260924 00:09:56.898794 7467 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:56.898850 7467 heartbeater.cc:511] Master 127.0.98.61:38143 requested a full tablet report, sending...
I20260924 00:09:56.899181 7100 ts_manager.cc:194] Registered new tserver with Master: 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
W20260924 00:09:56.983510 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:56.983736 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:56.983783 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:56.985044 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:56.985092 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:56.985133 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:09:56.986737 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41523
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:56.986930 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:56.987185 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:56.992285 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.992260 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:56.992264 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:56.992534 7473 server_base.cc:1034] running on GCE node
I20260924 00:09:56.992770 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:56.992993 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:56.994145 7473 hybrid_clock.cc:648] HybridClock initialized: now 1790208596994132 us; error 25 us; skew 500 ppm
Sep 24 00:09:56 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:56.996891 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:09:56.997403 7473 webserver.cc:460] Webserver started at http://127.0.98.2:41523/ using document root <none> and password file <none>
I20260924 00:09:56.997622 7473 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:56.997675 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:56.999083 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:56.999742 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:56.999935 7473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:09:57.000041 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "3230a4ecd5214bf7b0cd983f73d35dbf"
format_stamp: "Formatted at 2026-09-24 00:09:55 on dist-test-slave-3q8k"
server_key: "2d295b7938f3ca01df639fbc72e66c52"
server_key_iv: "e2b0aa74829351ac7bd8985477af6b91"
server_key_version: "encryptionkey@0"
I20260924 00:09:57.000370 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:57.024312 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:57.026763 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:57.026950 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:57.027259 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:57.027643 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:57.027686 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:57.027726 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:57.027755 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:57.035141 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34495
I20260924 00:09:57.035332 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34495 every 8 connection(s)
I20260924 00:09:57.035667 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:09:57.042795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7473
I20260924 00:09:57.042965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6832
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:09:57.048792 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34663 (local address 127.0.98.61:38143)
0924 00:09:57.037026 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.037028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.038162 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 00:09:57.038255 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.038256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.038267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.038353 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.039261 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.040019 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041547 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.041843 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041898 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.044831 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.044832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.044833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.044840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.047042 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047603 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.047604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.047605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.047665 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047992 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.047993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.047994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.048145 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.048202 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.048575 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.048671 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:09:57.048794 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41647 (local address 127.0.98.62:45797)
0924 00:09:57.036441 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.036443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.037019 (+ 576us) server_negotiation.cc:407] Connection header received
0924 00:09:57.038183 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.038184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.038197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.038231 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.039257 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.039765 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.040868 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.041081 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041174 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.044828 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.044830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.044831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.044840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.046944 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047597 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.047598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.047598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.047656 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047994 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.047995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.047996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.048143 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.048186 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.048562 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.048649 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:09:57.049372 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:41647)
0924 00:09:57.036953 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.036964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.038072 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.038380 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.038385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.038922 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.039133 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.039137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.039880 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.039881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.040797 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.040800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.041180 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.041310 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.041831 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.041839 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.044730 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.047044 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.047052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.047505 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.047723 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.047724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.047911 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.048232 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.048238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.048406 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.048887 (+ 481us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.049088 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 00:09:57.049398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36029
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45797
--webserver_port=44887
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:09:57.049901 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:57.049990 7603 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.050180 7603 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:57.050458 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.98.2:34663)
0924 00:09:57.038105 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.038115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.038173 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.038398 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.038399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.038911 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.039134 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.039139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.040117 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.040118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.041451 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.041452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.041927 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.041995 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.042448 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.042453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.044706 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.047098 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.047100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.047508 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.047742 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.047742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.047743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.047913 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.048236 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.048241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.048408 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.049357 (+ 949us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.050150 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":27,"thread_start_us":68,"threads_started":1}
I20260924 00:09:57.050690 6502 ts_manager.cc:194] Re-registered known tserver with Master: 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
I20260924 00:09:57.051247 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41647
I20260924 00:09:57.052616 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38143
I20260924 00:09:57.052677 7604 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.052760 7604 heartbeater.cc:511] Master 127.0.98.61:38143 requested a full tablet report, sending...
I20260924 00:09:57.053123 7100 ts_manager.cc:194] Registered new tserver with Master: 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
I20260924 00:09:57.054198 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43381 (local address 127.0.98.60:40151)
0924 00:09:57.036601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.036603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.036700 (+ 97us) server_negotiation.cc:407] Connection header received
0924 00:09:57.038189 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.038191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.038206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.038239 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.039262 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.040025 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.041940 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.042180 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.042241 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.050812 (+ 8571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.050815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.050815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.050825 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.052737 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.053150 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.053151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.053152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.053212 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.053472 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.053473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.053474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.053622 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.053698 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.053903 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.054011 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:09:57.054617 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.036403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.98.2:43381)
0924 00:09:57.036566 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.036619 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.038049 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.038379 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.038384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.038901 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.039134 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.039139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.040117 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.040118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.041436 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.041439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.042455 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.042456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.042521 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.042938 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.042943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.050689 (+ 7746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.052851 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.052852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.052852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.053063 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.053274 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.053275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.053275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.053342 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.053714 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.053716 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.053769 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.053856 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.053978 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":35,"thread_start_us":85,"threads_started":1}
I20260924 00:09:57.055045 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40151
I20260924 00:09:57.055104 7602 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.055155 7602 heartbeater.cc:511] Master 127.0.98.60:40151 requested a full tablet report, sending...
I20260924 00:09:57.055521 7286 ts_manager.cc:194] Registered new tserver with Master: 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
W20260924 00:09:57.133080 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:09:57.133288 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:09:57.133354 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:09:57.134475 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:09:57.134543 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:09:57.134570 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:09:57.136116 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44887
--tserver_master_addrs=127.0.98.62:45797,127.0.98.61:38143,127.0.98.60:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:09:57.136364 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:09:57.136626 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:09:57.141824 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:57.141959 7609 server_base.cc:1034] running on GCE node
W20260924 00:09:57.142035 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:09:57.141880 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:09:57.142273 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:09:57.142534 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:09:57.143710 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790208597143680 us; error 45 us; skew 500 ppm
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208597, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:09:57.146450 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:09:57.146900 7609 webserver.cc:460] Webserver started at http://127.0.98.3:44887/ using document root <none> and password file <none>
I20260924 00:09:57.147153 7609 fs_manager.cc:359] Metadata directory not provided
I20260924 00:09:57.147261 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:09:57.148648 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:57.149255 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:57.149498 7609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:57.149580 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
format_stamp: "Formatted at 2026-09-24 00:09:56 on dist-test-slave-3q8k"
server_key: "7cc0121a1e579632b11a48ab6c9cfce2"
server_key_iv: "487474143003f9baa025db5f6678ad49"
server_key_version: "encryptionkey@0"
I20260924 00:09:57.149901 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:09:57.159812 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.157207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.0.98.60:40151)
0924 00:09:57.157271 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.157273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.157286 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:09:57.157317 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.157318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.157346 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.157420 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.157571 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.158058 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.159005 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.159463 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.159593 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.159669 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.159704 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.159738 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:09:57.159812 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.157131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36502)
0924 00:09:57.157200 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.157211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.157260 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.157425 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.157426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.157428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.157483 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.157485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.158149 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.158150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.158899 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.158901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.159591 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.159592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.159650 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.159653 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.159689 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.159737 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:09:57.160208 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 00:09:57.160372 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321d9634ec5d44fc831f5360f9290504. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:57.160503 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:09:57.172632 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:09:57.175158 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:09:57.175386 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:09:57.175673 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:09:57.175998 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:09:57.176061 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:57.176105 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:09:57.176127 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:09:57.182124 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36029
I20260924 00:09:57.182181 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36029 every 8 connection(s)
I20260924 00:09:57.182590 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:09:57.185671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7609
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208597, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208597, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 00:09:57 dist-test-slave-3q8k krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208597, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:09:57.195744 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38443 (local address 127.0.98.62:45797)
0924 00:09:57.183881 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.183883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.183893 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:57.184354 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.184355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.184367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.184418 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.185142 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.185921 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187405 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.187732 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187778 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.191965 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.191966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.191967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.191974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.193772 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.194267 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.194268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.194269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.194329 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.194874 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.194875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.194876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.195018 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.195075 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.195544 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.195627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:57.196352 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:38443)
0924 00:09:57.183628 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.183671 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.184236 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.184498 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.184502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.184826 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.185029 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.185033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.186018 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.186019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.187300 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.187303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.188129 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.188129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.188219 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.188857 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.188868 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.191852 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.193855 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.193862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.193865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.194186 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.194446 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.194447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.194448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.194778 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.195127 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.195134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.195363 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.195845 (+ 482us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.196093 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 00:09:57.197052 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45797
I20260924 00:09:57.197160 7739 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.197221 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46313 (local address 127.0.98.60:40151)
0924 00:09:57.183559 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.183561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.183894 (+ 333us) server_negotiation.cc:407] Connection header received
0924 00:09:57.184394 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.184395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.184412 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.184442 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.185171 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.185921 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187397 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.187708 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187752 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.192931 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.192932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.192932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.192940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.194860 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.196293 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.196294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.196294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.196349 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.196682 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.196683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.196684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.196823 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.196876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.197052 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.197131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:09:57.197377 7739 heartbeater.cc:511] Master 127.0.98.62:45797 requested a full tablet report, sending...
I20260924 00:09:57.197526 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.98.3:46313)
0924 00:09:57.183789 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.183833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.184280 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.184511 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.184512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.184848 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.185055 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.185057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.186238 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.186239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.187297 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.187300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.187841 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.187985 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.189327 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.189332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.192517 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.195123 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.195124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.195125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.196201 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.196526 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.196527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.196528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.196587 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.196906 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.196907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.196945 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.197321 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.197424 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260924 00:09:57.197779 6502 ts_manager.cc:194] Re-registered known tserver with Master: 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029)
I20260924 00:09:57.197829 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40151
I20260924 00:09:57.197868 7738 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.197924 7738 heartbeater.cc:511] Master 127.0.98.60:40151 requested a full tablet report, sending...
I20260924 00:09:57.198174 7286 ts_manager.cc:194] Registered new tserver with Master: 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029)
I20260924 00:09:57.198253 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38443
I20260924 00:09:57.199700 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41565 (local address 127.0.98.61:38143)
0924 00:09:57.183803 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.183805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.183956 (+ 151us) server_negotiation.cc:407] Connection header received
0924 00:09:57.184394 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.184395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.184408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.184442 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.185203 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.185840 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.187880 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.188161 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.188709 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.193877 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.193878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.193878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.193885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.195514 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.198816 (+ 3302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.198818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.198818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.198874 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.199083 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.199084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.199084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.199248 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.199304 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.199484 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.199591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:09:57.199884 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.186774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.0.98.62:45797)
0924 00:09:57.187761 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.187763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.187769 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:09:57.187789 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.187790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.187801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.187859 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.188721 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.189479 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.190708 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.191092 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.191161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.192716 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.192717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.192718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.192723 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.197163 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.198586 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.198587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.198587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.198645 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.198865 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.198865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.198866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.198994 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.199141 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.199702 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.199787 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":92,"threads_started":1}
I20260924 00:09:57.199924 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.186951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.98.60:40151)
0924 00:09:57.187050 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.187052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.187702 (+ 650us) server_negotiation.cc:407] Connection header received
0924 00:09:57.188366 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.188367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.188381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.188430 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.189672 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.190434 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.191499 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.191808 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.191848 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.193886 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.193888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.193888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.193895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.196022 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.196398 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.196399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.196399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.196447 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.199268 (+ 2821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.199270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.199270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.199419 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.199535 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.199745 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.199837 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":55,"threads_started":1}
I20260924 00:09:57.200039 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.183462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.98.3:41565)
0924 00:09:57.183906 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.183916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.184279 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.184936 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.184938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.185084 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:09:57.185143 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.185145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.185976 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.185977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.187785 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.187788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.188801 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.188801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.188878 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.189327 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:09:57.189332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:09:57.193456 (+ 4124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:09:57.198527 (+ 5071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.198528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.198529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.198726 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.198946 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:09:57.198947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:09:57.198947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:09:57.199002 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:09:57.199308 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:09:57.199309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:09:57.199349 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.199571 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.199927 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":1056,"thread_start_us":68,"threads_started":1}
I20260924 00:09:57.200461 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38143
I20260924 00:09:57.200511 7740 heartbeater.cc:464] Registering TS with master...
I20260924 00:09:57.200582 7740 heartbeater.cc:511] Master 127.0.98.61:38143 requested a full tablet report, sending...
I20260924 00:09:57.201066 7100 ts_manager.cc:194] Registered new tserver with Master: 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029)
I20260924 00:09:57.202268 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.188007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.0.98.61:38143)
0924 00:09:57.189412 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.189415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.189422 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:09:57.189450 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.189451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.189469 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.189543 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.191055 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.192063 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.193518 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.195466 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.195540 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.198261 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.198263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.198263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.198272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.201020 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.201403 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.201404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.201405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.201475 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.201785 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.201786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.201786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.201880 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.201945 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.202119 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.202179 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":54,"threads_started":1}
I20260924 00:09:57.201651 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:57.203685 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:57.204480 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: true } } }
I20260924 00:09:57.204583 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:57.205736 7061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: attempting to promote NON_VOTER fc409f33c33c4e2a84d324fc033beb19 to VOTER
I20260924 00:09:57.205888 7061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } }
I20260924 00:09:57.206264 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 LEARNER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 00:09:57.206379 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Refusing update from remote peer 321d9634ec5d44fc831f5360f9290504: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 00:09:57.206601 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:09:57.206687 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:09:57.207453 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.207571 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.207636 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208325 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208396 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:57.208459 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208510 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:57.208549 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208590 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:57.208626 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208694 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:09:57.208684 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "321d9634ec5d44fc831f5360f9290504" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:57.208760 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:09:57.214342 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.210714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34495 (local address 127.0.0.1:56246)
0924 00:09:57.210892 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.210903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.210941 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.211134 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.211135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.211137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.211189 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.211191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212272 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.212273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.213239 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.213241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.214139 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.214139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.214194 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.214197 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.214235 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.214270 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260924 00:09:57.214360 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.210803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.98.2:34495)
0924 00:09:57.210982 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.210987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.210998 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:57.211039 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.211041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.211058 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.211137 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.211366 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212174 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.213337 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.214001 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.214141 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.214219 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.214247 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.214281 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260924 00:09:57.215121 7536 tablet_service.cc:1467] Processing CreateTablet for tablet e3d348a6a0024429a8810d20172faed4 (DEFAULT_TABLE table=test-table [id=6f7864e01dd4432d9a7157521004e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:57.215440 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.210669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45075 (local address 127.0.0.1:41510)
0924 00:09:57.210892 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.210902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.210963 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.211271 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.211272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.211274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.211338 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.211340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212725 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.212726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.213995 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.213998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.215119 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.215120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.215177 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.215180 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.215307 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.215348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":26}
I20260924 00:09:57.215559 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d348a6a0024429a8810d20172faed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:57.215757 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.210765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.0.98.1:45075)
0924 00:09:57.211085 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.211088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.211098 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:57.211133 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.211135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.211148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.211412 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.211546 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212630 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.214095 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.215026 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.215097 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.215166 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.215325 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.215404 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":90,"threads_started":1}
I20260924 00:09:57.216333 7399 tablet_service.cc:1467] Processing CreateTablet for tablet e3d348a6a0024429a8810d20172faed4 (DEFAULT_TABLE table=test-table [id=6f7864e01dd4432d9a7157521004e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:57.216658 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d348a6a0024429a8810d20172faed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:57.219136 7767 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Bootstrap starting.
I20260924 00:09:57.219347 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.210767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36029 (local address 127.0.0.1:35934)
0924 00:09:57.211248 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.211258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.211296 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.211574 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.211575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.211578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.211611 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.211612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212785 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.212786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.213672 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.213674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.214560 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.214561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.214621 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.214626 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.219152 (+ 4526us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.219203 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":88,"threads_started":1}
I20260924 00:09:57.219827 7767 tablet_bootstrap.cc:654] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:57.219867 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.211105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.0.98.3:36029)
0924 00:09:57.211405 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.211409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.211418 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:57.211463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.211465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.211479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.211678 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.211727 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.212721 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.213766 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.214492 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.214563 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.214631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.219543 (+ 4912us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.219599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":78,"threads_started":1}
I20260924 00:09:57.220314 7768 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Bootstrap starting.
I20260924 00:09:57.220690 7767 log.cc:824] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:57.221048 7768 tablet_bootstrap.cc:654] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:57.221016 7672 tablet_service.cc:1467] Processing CreateTablet for tablet e3d348a6a0024429a8810d20172faed4 (DEFAULT_TABLE table=test-table [id=6f7864e01dd4432d9a7157521004e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:57.221381 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d348a6a0024429a8810d20172faed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:57.221385 7768 log.cc:824] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:57.221596 7767 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: No bootstrap required, opened a new log
I20260924 00:09:57.221664 7767 ts_tablet_manager.cc:1397] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:09:57.222160 7768 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: No bootstrap required, opened a new log
I20260924 00:09:57.222242 7768 ts_tablet_manager.cc:1397] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:09:57.223783 7768 raft_consensus.cc:348] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.223945 7768 raft_consensus.cc:374] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:57.223989 7768 raft_consensus.cc:729] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c93a21847b4defa92167f368274440, State: Initialized, Role: FOLLOWER
I20260924 00:09:57.224143 7768 consensus_queue.cc:260] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.224403 7768 ts_tablet_manager.cc:1428] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:57.224491 7466 heartbeater.cc:503] Master 127.0.98.62:45797 was elected leader, sending a full tablet report...
I20260924 00:09:57.224956 7772 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Bootstrap starting.
I20260924 00:09:57.225713 7772 tablet_bootstrap.cc:654] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:57.224396 7767 raft_consensus.cc:348] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.225898 7767 raft_consensus.cc:374] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:57.225955 7767 raft_consensus.cc:729] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3230a4ecd5214bf7b0cd983f73d35dbf, State: Initialized, Role: FOLLOWER
I20260924 00:09:57.226042 7772 log.cc:824] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Log is configured to *not* fsync() on all Append() calls
I20260924 00:09:57.226111 7767 consensus_queue.cc:260] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.226361 7767 ts_tablet_manager.cc:1428] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260924 00:09:57.226575 7603 heartbeater.cc:503] Master 127.0.98.62:45797 was elected leader, sending a full tablet report...
I20260924 00:09:57.227254 7772 tablet_bootstrap.cc:492] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: No bootstrap required, opened a new log
I20260924 00:09:57.227339 7772 ts_tablet_manager.cc:1397] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:57.229183 7772 raft_consensus.cc:348] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.229296 7772 raft_consensus.cc:374] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:57.229317 7772 raft_consensus.cc:729] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ea89f0b12c40f09fae826ead7b8bbc, State: Initialized, Role: FOLLOWER
I20260924 00:09:57.229425 7772 consensus_queue.cc:260] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.229632 7772 ts_tablet_manager.cc:1428] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:09:57.229704 7739 heartbeater.cc:503] Master 127.0.98.62:45797 was elected leader, sending a full tablet report...
W20260924 00:09:57.287015 7605 tablet.cc:2369] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:57.291724 7771 raft_consensus.cc:482] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:57.291843 7771 raft_consensus.cc:504] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.292408 7771 leader_election.cc:288] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029), 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
I20260924 00:09:57.296597 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.292613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36029 (local address 127.0.98.1:54249)
0924 00:09:57.292676 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.292684 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.292726 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.292886 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.292887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.292889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.292999 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.293000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.293933 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.293934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.295262 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.295264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.296287 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.296288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.296393 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.296396 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.296460 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.296490 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":51}
I20260924 00:09:57.296597 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.292679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43365 (local address 127.0.98.2:34495)
0924 00:09:57.292718 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.292720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.292729 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:57.292785 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.292786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.292799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.292862 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.293081 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.293815 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.295482 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.296184 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.296277 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.296300 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.296470 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.296524 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:09:57.296638 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.292642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54249 (local address 127.0.98.3:36029)
0924 00:09:57.292686 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.292688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.292753 (+ 65us) server_negotiation.cc:407] Connection header received
0924 00:09:57.292785 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.292786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.292799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.292863 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:57.293081 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.293846 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.295559 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.296184 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.296277 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.296289 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.296524 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.296566 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:09:57.296608 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.292572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34495 (local address 127.0.98.1:43365)
0924 00:09:57.292662 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:57.292676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:57.292726 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:57.292886 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:57.292887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:57.292889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:57.292991 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.292993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.293933 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.293934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.295260 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:57.295264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.296287 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.296288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:57.296369 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.296374 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:57.296438 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:09:57.296487 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":102}
I20260924 00:09:57.297192 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d348a6a0024429a8810d20172faed4" candidate_uuid: "25c93a21847b4defa92167f368274440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" is_pre_election: true
I20260924 00:09:57.297384 7556 raft_consensus.cc:2454] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25c93a21847b4defa92167f368274440 in term 0.
I20260924 00:09:57.297263 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d348a6a0024429a8810d20172faed4" candidate_uuid: "25c93a21847b4defa92167f368274440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" is_pre_election: true
I20260924 00:09:57.297492 7692 raft_consensus.cc:2454] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25c93a21847b4defa92167f368274440 in term 0.
I20260924 00:09:57.297631 7352 leader_election.cc:302] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25c93a21847b4defa92167f368274440, 3230a4ecd5214bf7b0cd983f73d35dbf; no voters:
I20260924 00:09:57.297753 7771 raft_consensus.cc:2781] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:57.297820 7771 raft_consensus.cc:482] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:57.297884 7771 raft_consensus.cc:3037] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:57.298622 7771 raft_consensus.cc:504] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.298791 7771 leader_election.cc:288] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [CANDIDATE]: Term 1 election: Requested vote from peers 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029), 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2:34495)
I20260924 00:09:57.298950 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d348a6a0024429a8810d20172faed4" candidate_uuid: "25c93a21847b4defa92167f368274440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
I20260924 00:09:57.299007 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d348a6a0024429a8810d20172faed4" candidate_uuid: "25c93a21847b4defa92167f368274440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf"
I20260924 00:09:57.299039 7692 raft_consensus.cc:3037] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:57.299077 7556 raft_consensus.cc:3037] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:57.300041 7556 raft_consensus.cc:2454] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25c93a21847b4defa92167f368274440 in term 1.
I20260924 00:09:57.300041 7692 raft_consensus.cc:2454] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25c93a21847b4defa92167f368274440 in term 1.
I20260924 00:09:57.300247 7352 leader_election.cc:302] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25c93a21847b4defa92167f368274440, 3230a4ecd5214bf7b0cd983f73d35dbf; no voters:
I20260924 00:09:57.300498 7771 raft_consensus.cc:2781] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:57.300699 7771 raft_consensus.cc:686] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 1 LEADER]: Becoming Leader. State: Replica: 25c93a21847b4defa92167f368274440, State: Running, Role: LEADER
I20260924 00:09:57.300810 7771 consensus_queue.cc:237] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } }
I20260924 00:09:57.301611 6502 catalog_manager.cc:5495] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25c93a21847b4defa92167f368274440 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "25c93a21847b4defa92167f368274440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:57.307876 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.305536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.0.98.1:45075)
0924 00:09:57.305585 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.305587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.305595 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:57.305671 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.305671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.305687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.305744 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:57.305890 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.306332 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.307120 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.307370 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.307423 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.307568 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:57.307659 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:57.307715 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.307759 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.307774 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:09:57.309733 7556 raft_consensus.cc:1261] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Refusing update from remote peer 25c93a21847b4defa92167f368274440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:57.309756 7692 raft_consensus.cc:1261] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Refusing update from remote peer 25c93a21847b4defa92167f368274440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:57.310138 7771 consensus_queue.cc:1035] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:57.310289 7776 consensus_queue.cc:1035] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:57.315447 7781 mvcc.cc:205] Tried to move back new op lower bound from 7332694414577639424 to 7332694414544568320. Current Snapshot: MvccSnapshot[applied={T|T < 7332694414577639424 or (T in {7332694414577639424})}]
I20260924 00:09:57.316162 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:57.316251 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:57.317818 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 321d9634ec5d44fc831f5360f9290504: Sending DeleteTablet for 3 replicas of tablet e3d348a6a0024429a8810d20172faed4
I20260924 00:09:57.318162 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d348a6a0024429a8810d20172faed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56246
I20260924 00:09:57.318150 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d348a6a0024429a8810d20172faed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41510
I20260924 00:09:57.318174 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d348a6a0024429a8810d20172faed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35934
I20260924 00:09:57.318420 7789 tablet_replica.cc:330] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: stopping tablet replica
I20260924 00:09:57.318537 7789 raft_consensus.cc:2229] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:57.318655 7789 raft_consensus.cc:2258] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:57.318828 7788 tablet_replica.cc:330] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: stopping tablet replica
I20260924 00:09:57.319084 7790 tablet_replica.cc:330] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: stopping tablet replica
I20260924 00:09:57.319178 7790 raft_consensus.cc:2229] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:57.319293 7790 raft_consensus.cc:2258] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:57.319406 7788 raft_consensus.cc:2229] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:57.319468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6470
I20260924 00:09:57.319530 7788 raft_consensus.cc:2258] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:57.320170 7788 ts_tablet_manager.cc:1905] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:57.322208 7788 ts_tablet_manager.cc:1918] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:57.322288 7788 log.cc:1196] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/e3d348a6a0024429a8810d20172faed4
I20260924 00:09:57.322546 7788 ts_tablet_manager.cc:1939] T e3d348a6a0024429a8810d20172faed4 P 25c93a21847b4defa92167f368274440: Deleting consensus metadata
W20260924 00:09:57.325927 7352 connection.cc:535] server connection from 127.0.0.1:41510 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:09:57.326560 7789 ts_tablet_manager.cc:1905] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:57.327057 7790 ts_tablet_manager.cc:1905] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:57.328944 7790 ts_tablet_manager.cc:1918] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:57.329047 7790 log.cc:1196] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/e3d348a6a0024429a8810d20172faed4
I20260924 00:09:57.329334 7790 ts_tablet_manager.cc:1939] T e3d348a6a0024429a8810d20172faed4 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting consensus metadata
W20260924 00:09:57.329829 7625 connection.cc:441] server connection from 127.0.0.1:35934 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35934 (request call id 1) could send its response
I20260924 00:09:57.330657 7789 ts_tablet_manager.cc:1918] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:57.330744 7789 log.cc:1196] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/e3d348a6a0024429a8810d20172faed4
I20260924 00:09:57.331035 7789 ts_tablet_manager.cc:1939] T e3d348a6a0024429a8810d20172faed4 P 3230a4ecd5214bf7b0cd983f73d35dbf: Deleting consensus metadata
W20260924 00:09:57.331414 7491 connection.cc:441] server connection from 127.0.0.1:56246 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56246 (request call id 1) could send its response
I20260924 00:09:57.334398 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.319012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.98.60:40151)
0924 00:09:57.319064 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.319066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.326726 (+ 7660us) server_negotiation.cc:407] Connection header received
0924 00:09:57.326836 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.326837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.326852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.326883 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.327791 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.328295 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.329092 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.329415 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.329465 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.331132 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.331133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.331133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.331141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.333121 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.333534 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.333536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.333536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.333592 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.333765 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.333765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.333766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.333891 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.333980 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.334218 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.334301 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:09:57.340018 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.331617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.60:40151)
0924 00:09:57.331664 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.331666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.332136 (+ 470us) server_negotiation.cc:407] Connection header received
0924 00:09:57.332228 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.332229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.332240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.332274 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.333012 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.333766 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.335094 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.335405 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.335471 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.336813 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.336815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.336815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.336822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.338506 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.338968 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.338969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.338975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.339067 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.339351 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.339352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.339352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.339503 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.339588 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.339832 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.339914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:09:57.340250 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:57.331483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.0.98.61:38143)
0924 00:09:57.331529 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:57.331531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:57.331820 (+ 289us) server_negotiation.cc:407] Connection header received
0924 00:09:57.331875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:57.331876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:57.331891 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:57.331961 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:09:57.333032 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.333944 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.335427 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:57.335712 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:57.335779 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:57.337157 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:09:57.337158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:09:57.337159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:09:57.337166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:09:57.338828 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.339466 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.339467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.339467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.339519 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:09:57.339737 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:09:57.339738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:09:57.339738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:09:57.339855 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:09:57.339961 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:57.340067 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 00:09:57.340165 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:09:58.314951 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.314681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:56667)
0924 00:09:58.314811 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.314891 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
W20260924 00:09:58.315263 7603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.316563 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.316310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:48093)
0924 00:09:58.316419 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.316485 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260924 00:09:58.316644 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.316339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:38671)
0924 00:09:58.316525 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.316597 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
W20260924 00:09:58.316751 7739 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
W20260924 00:09:58.316804 7466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.817790 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 321d9634ec5d44fc831f5360f9290504)
I20260924 00:09:58.818001 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } }
I20260924 00:09:58.818310 7801 leader_election.cc:288] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797), 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61:38143)
I20260924 00:09:58.818781 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 321d9634ec5d44fc831f5360f9290504)
I20260924 00:09:58.818810 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.818498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40312)
0924 00:09:58.818716 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.818765 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":44,"threads_started":1}
I20260924 00:09:58.818904 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } }
W20260924 00:09:58.818976 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.819468 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.819332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40328)
0924 00:09:58.819398 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.819433 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 00:09:58.819577 7270 leader_election.cc:334] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.819736 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.819255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40322)
0924 00:09:58.819412 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.819484 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":55,"threads_started":1}
I20260924 00:09:58.819766 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797), fc409f33c33c4e2a84d324fc033beb19 (127.0.98.60:40151)
W20260924 00:09:58.819825 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.821202 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.821101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40330)
0924 00:09:58.821154 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.821178 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 00:09:58.821271 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.821846 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.818498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38143 (local address 127.0.0.1:60502)
0924 00:09:58.818608 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.818620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:58.818705 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:58.819013 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:58.819015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:58.819017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:58.819074 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.819076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.820055 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.820056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.820908 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.820910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.821625 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.821626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.821704 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.821708 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:58.821737 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 00:09:58.821778 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260924 00:09:58.822089 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.818598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.61:38143)
0924 00:09:58.818778 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:58.818781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:58.818791 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:09:58.818821 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:58.818822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:58.818834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:58.818954 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:58.819166 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.819982 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.821019 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.821529 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.821962 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.821977 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:58.821996 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:09:58.822033 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260924 00:09:58.822248 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc409f33c33c4e2a84d324fc033beb19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" is_pre_election: true
I20260924 00:09:58.822335 7110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc409f33c33c4e2a84d324fc033beb19 in term 1.
I20260924 00:09:58.822481 7271 leader_election.cc:302] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ad2d49f51c4c7f87c65e4cde39a77a, fc409f33c33c4e2a84d324fc033beb19; no voters: 321d9634ec5d44fc831f5360f9290504
I20260924 00:09:58.822638 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:09:58.822677 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 321d9634ec5d44fc831f5360f9290504)
I20260924 00:09:58.822697 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:09:58.823460 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.819491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40151 (local address 127.0.0.1:36526)
0924 00:09:58.819588 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.819599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:58.819649 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:58.820170 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:58.820171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:58.820173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:58.820209 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.820211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.820889 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.820890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.822531 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.822534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.823188 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.823189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.823278 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.823281 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:09:58.823341 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 00:09:58.823386 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":38,"threads_started":1}
I20260924 00:09:58.823465 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.819665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.60:40151)
0924 00:09:58.820023 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:58.820025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:58.820034 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:58.820055 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:58.820056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:58.820065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:58.820145 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:58.820309 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.820796 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.822651 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.823098 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.823187 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.823197 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:58.823372 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 00:09:58.823410 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":79,"threads_started":1}
I20260924 00:09:58.823494 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } }
I20260924 00:09:58.823659 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc409f33c33c4e2a84d324fc033beb19" is_pre_election: true
I20260924 00:09:58.823783 7296 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69ad2d49f51c4c7f87c65e4cde39a77a in current term 2: Already voted for candidate fc409f33c33c4e2a84d324fc033beb19 in this term.
I20260924 00:09:58.823791 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc409f33c33c4e2a84d324fc033beb19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a"
I20260924 00:09:58.823829 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.823718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40348)
0924 00:09:58.823755 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.823778 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 00:09:58.823860 7110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:09:58.824162 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.824051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40350)
0924 00:09:58.824111 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.824141 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:09:58.824270 7270 leader_election.cc:334] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
I20260924 00:09:58.824633 7110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc409f33c33c4e2a84d324fc033beb19 in term 2.
I20260924 00:09:58.824838 7086 leader_election.cc:302] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69ad2d49f51c4c7f87c65e4cde39a77a; no voters: 321d9634ec5d44fc831f5360f9290504, fc409f33c33c4e2a84d324fc033beb19
I20260924 00:09:58.824935 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:09:58.824990 7271 leader_election.cc:302] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69ad2d49f51c4c7f87c65e4cde39a77a, fc409f33c33c4e2a84d324fc033beb19; no voters: 321d9634ec5d44fc831f5360f9290504
I20260924 00:09:58.823660 7801 leader_election.cc:288] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [CANDIDATE]: Term 2 election: Requested vote from peers 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797), 69ad2d49f51c4c7f87c65e4cde39a77a (127.0.98.61:38143)
I20260924 00:09:58.825163 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:09:58.825273 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [term 2 LEADER]: Becoming Leader. State: Replica: fc409f33c33c4e2a84d324fc033beb19, State: Running, Role: LEADER
I20260924 00:09:58.825353 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } }
I20260924 00:09:58.825801 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc409f33c33c4e2a84d324fc033beb19. Latest consensus state: current_term: 2 leader_uuid: "fc409f33c33c4e2a84d324fc033beb19" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:58.825894 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:58.826068 7813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:09:58.826615 7813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:09:58.826678 7813 catalog_manager.cc:1259] Loaded cluster ID: 57ccc3e6d83141088e7a54365b4a6592
I20260924 00:09:58.826696 7813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:09:58.826879 7813 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:09:58.826983 7813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Loaded TSK: 0
I20260924 00:09:58.827122 7813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:09:58.879287 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:09:58.880957 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:09:58.882112 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [term 2 FOLLOWER]: Refusing update from remote peer fc409f33c33c4e2a84d324fc033beb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 00:09:58.882125 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.882005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40382)
0924 00:09:58.882067 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.882096 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 00:09:58.882460 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.882349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.0.1:40386)
0924 00:09:58.882388 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.882411 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 00:09:58.882403 7801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 00:09:58.882552 7270 consensus_peers.cc:487] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 -> Peer 321d9634ec5d44fc831f5360f9290504 (127.0.98.62:45797): Couldn't send request to peer 321d9634ec5d44fc831f5360f9290504. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:09:58.883126 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc409f33c33c4e2a84d324fc033beb19. Latest consensus state: current_term: 2 leader_uuid: "fc409f33c33c4e2a84d324fc033beb19" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:58.883270 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:09:58.883715 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fc409f33c33c4e2a84d324fc033beb19" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:58.883803 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:09:58.883950 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fc409f33c33c4e2a84d324fc033beb19" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "321d9634ec5d44fc831f5360f9290504" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45797 } } peers { permanent_uuid: "69ad2d49f51c4c7f87c65e4cde39a77a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38143 } attrs { promote: false } } peers { permanent_uuid: "fc409f33c33c4e2a84d324fc033beb19" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40151 } attrs { promote: false } } }
I20260924 00:09:58.884054 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad2d49f51c4c7f87c65e4cde39a77a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:09:58.884107 7815 mvcc.cc:205] Tried to move back new op lower bound from 7332694421018705920 to 7332694420789571584. Current Snapshot: MvccSnapshot[applied={T|T < 7332694421018705920 or (T in {7332694421018705920})}]
I20260924 00:09:58.888844 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34495 (local address 127.0.0.1:56262)
0924 00:09:58.885657 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.885667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:58.885708 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:58.885984 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:58.885985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:58.885987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:58.886042 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.886043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.887098 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.887099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.887936 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.887938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.888630 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.888691 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.888696 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:58.888724 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 00:09:58.888763 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":4}
I20260924 00:09:58.889112 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.98.2:34495)
0924 00:09:58.885831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:58.885833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:58.885842 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:58.885867 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:58.885868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:58.885878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:58.886106 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:58.886139 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.887008 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888024 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.888546 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888975 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.888991 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:58.889009 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 00:09:58.889044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260924 00:09:58.889246 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36029 (local address 127.0.0.1:35946)
0924 00:09:58.885813 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.885825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:58.885877 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:58.886216 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:58.886217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:58.886218 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:58.886252 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.886252 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.887022 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.887023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.888269 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.888271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.889024 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.889025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.889088 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.889090 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:09:58.889132 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 00:09:58.889171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":78}
I20260924 00:09:58.889314 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 (DEFAULT_TABLE table=test-table [id=4975d0411efb4017ae8689c40d564835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:58.889286 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.0.98.3:36029)
0924 00:09:58.886072 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:58.886075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:58.886083 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:09:58.886110 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:58.886111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:58.886120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:58.886192 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:58.886346 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.886929 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888364 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.888825 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888922 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.889158 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:58.889177 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:09:58.889215 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260924 00:09:58.889442 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a7bdd5dd34fc7a090f7908969a706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:58.889528 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45075 (local address 127.0.0.1:41530)
0924 00:09:58.885923 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.885933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:58.885974 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:58.886521 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:58.886522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:58.886523 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:58.886561 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.886562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.887449 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.887450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.888484 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:58.888485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.889303 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.889304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:58.889360 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.889364 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:58.889411 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 00:09:58.889466 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
I20260924 00:09:58.889549 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 (DEFAULT_TABLE table=test-table [id=4975d0411efb4017ae8689c40d564835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:58.889655 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a7bdd5dd34fc7a090f7908969a706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:58.889775 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.885966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.1:45075)
0924 00:09:58.886074 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:58.886076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:58.886083 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:09:58.886110 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:58.886111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:58.886122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:58.886191 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:58.886645 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.887358 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.888547 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:58.889192 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:58.889258 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:58.889272 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:58.889673 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 00:09:58.889713 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260924 00:09:58.889941 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 (DEFAULT_TABLE table=test-table [id=4975d0411efb4017ae8689c40d564835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:09:58.890043 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a7bdd5dd34fc7a090f7908969a706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:09:58.890929 7823 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: Bootstrap starting.
I20260924 00:09:58.891196 7824 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Bootstrap starting.
I20260924 00:09:58.891465 7823 tablet_bootstrap.cc:654] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:58.891510 7825 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440: Bootstrap starting.
I20260924 00:09:58.891570 7824 tablet_bootstrap.cc:654] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:58.891903 7825 tablet_bootstrap.cc:654] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440: Neither blocks nor log segments found. Creating new log.
I20260924 00:09:58.892218 7824 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: No bootstrap required, opened a new log
I20260924 00:09:58.892230 7823 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: No bootstrap required, opened a new log
I20260924 00:09:58.892279 7823 ts_tablet_manager.cc:1397] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:58.892293 7824 ts_tablet_manager.cc:1397] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:09:58.892407 7824 raft_consensus.cc:348] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.892499 7825 tablet_bootstrap.cc:492] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440: No bootstrap required, opened a new log
I20260924 00:09:58.892503 7824 raft_consensus.cc:374] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:58.892437 7823 raft_consensus.cc:348] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.892525 7824 raft_consensus.cc:729] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ea89f0b12c40f09fae826ead7b8bbc, State: Initialized, Role: FOLLOWER
I20260924 00:09:58.892529 7823 raft_consensus.cc:374] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:58.892539 7825 ts_tablet_manager.cc:1397] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:58.892554 7823 raft_consensus.cc:729] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3230a4ecd5214bf7b0cd983f73d35dbf, State: Initialized, Role: FOLLOWER
I20260924 00:09:58.892568 7824 consensus_queue.cc:260] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.892619 7823 consensus_queue.cc:260] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.892704 7824 ts_tablet_manager.cc:1428] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:09:58.892669 7825 raft_consensus.cc:348] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.892743 7825 raft_consensus.cc:374] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:09:58.892774 7825 raft_consensus.cc:729] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c93a21847b4defa92167f368274440, State: Initialized, Role: FOLLOWER
I20260924 00:09:58.892777 7823 ts_tablet_manager.cc:1428] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:09:58.892832 7825 consensus_queue.cc:260] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:58.893363 7825 ts_tablet_manager.cc:1428] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:09:58.893473 7602 heartbeater.cc:503] Master 127.0.98.60:40151 was elected leader, sending a full tablet report...
I20260924 00:09:58.893498 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.893016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:47647)
0924 00:09:58.893417 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.893466 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":79,"threads_started":1}
I20260924 00:09:58.893787 7738 heartbeater.cc:503] Master 127.0.98.60:40151 was elected leader, sending a full tablet report...
I20260924 00:09:58.893894 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.893741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:34053)
0924 00:09:58.893777 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.893799 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:09:58.893950 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.893199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:33989)
0924 00:09:58.893880 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.893914 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":67,"threads_started":1}
I20260924 00:09:58.894130 7465 heartbeater.cc:503] Master 127.0.98.60:40151 was elected leader, sending a full tablet report...
I20260924 00:09:58.894630 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.894539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:57393)
0924 00:09:58.894572 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.894601 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 00:09:58.894698 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.893928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:50883)
0924 00:09:58.894629 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.894661 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":72,"threads_started":1}
I20260924 00:09:58.895071 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:58.894964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:39949)
0924 00:09:58.895023 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:58.895048 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 00:09:58.934656 7741 tablet.cc:2369] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:09:59.093137 7830 raft_consensus.cc:482] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:09:59.093243 7830 raft_consensus.cc:504] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:59.093755 7830 leader_election.cc:288] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029), 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
I20260924 00:09:59.096850 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.094075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50469 (local address 127.0.98.3:36029)
0924 00:09:59.094144 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:59.094147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:59.094158 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:09:59.094190 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:59.094191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:59.094203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:59.094281 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:59.094516 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.094986 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096045 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.096491 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096583 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:59.096595 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:59.096748 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:09:59.096782 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:09:59.096907 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.094209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49543 (local address 127.0.98.1:45075)
0924 00:09:59.094280 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:59.094282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:59.094384 (+ 102us) server_negotiation.cc:407] Connection header received
0924 00:09:59.094450 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:59.094451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:59.094468 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:59.094543 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:09:59.094646 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.095108 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096109 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.096523 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096617 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:59.096628 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:59.096786 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:09:59.096821 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:09:59.096910 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.094126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45075 (local address 127.0.98.2:49543)
0924 00:09:59.094315 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.094325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:59.094396 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:59.094545 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:59.094546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:59.094547 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:59.094595 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:59.094597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.095265 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.095266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:59.095994 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:59.095996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.096621 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:59.096687 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:59.096692 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:09:59.096776 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:09:59.096815 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260924 00:09:59.096922 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.093980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36029 (local address 127.0.98.2:50469)
0924 00:09:59.094037 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.094052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:09:59.094106 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:09:59.094305 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:09:59.094307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:09:59.094311 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:09:59.094392 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:59.094394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.095075 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.095076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:59.095937 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:09:59.095939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.096583 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.096584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:09:59.096656 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:59.096660 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:09:59.096731 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:09:59.096774 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 00:09:59.097157 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a7bdd5dd34fc7a090f7908969a706" candidate_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25c93a21847b4defa92167f368274440" is_pre_election: true
I20260924 00:09:59.097247 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a7bdd5dd34fc7a090f7908969a706" candidate_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" is_pre_election: true
I20260924 00:09:59.097323 7692 raft_consensus.cc:2454] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3230a4ecd5214bf7b0cd983f73d35dbf in term 0.
I20260924 00:09:59.097343 7419 raft_consensus.cc:2454] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3230a4ecd5214bf7b0cd983f73d35dbf in term 0.
I20260924 00:09:59.097486 7492 leader_election.cc:302] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3230a4ecd5214bf7b0cd983f73d35dbf, 54ea89f0b12c40f09fae826ead7b8bbc; no voters:
I20260924 00:09:59.097638 7830 raft_consensus.cc:2781] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:09:59.097681 7830 raft_consensus.cc:482] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:09:59.097702 7830 raft_consensus.cc:3037] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:59.098346 7830 raft_consensus.cc:504] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:59.098485 7830 leader_election.cc:288] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [CANDIDATE]: Term 1 election: Requested vote from peers 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029), 25c93a21847b4defa92167f368274440 (127.0.98.1:45075)
I20260924 00:09:59.098644 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a7bdd5dd34fc7a090f7908969a706" candidate_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea89f0b12c40f09fae826ead7b8bbc"
I20260924 00:09:59.098654 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a7bdd5dd34fc7a090f7908969a706" candidate_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25c93a21847b4defa92167f368274440"
I20260924 00:09:59.098717 7419 raft_consensus.cc:3037] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:59.098713 7692 raft_consensus.cc:3037] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:09:59.099352 7692 raft_consensus.cc:2454] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3230a4ecd5214bf7b0cd983f73d35dbf in term 1.
I20260924 00:09:59.099516 7419 raft_consensus.cc:2454] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3230a4ecd5214bf7b0cd983f73d35dbf in term 1.
I20260924 00:09:59.099581 7492 leader_election.cc:302] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3230a4ecd5214bf7b0cd983f73d35dbf, 54ea89f0b12c40f09fae826ead7b8bbc; no voters:
I20260924 00:09:59.099669 7830 raft_consensus.cc:2781] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:09:59.099843 7830 raft_consensus.cc:686] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 LEADER]: Becoming Leader. State: Replica: 3230a4ecd5214bf7b0cd983f73d35dbf, State: Running, Role: LEADER
I20260924 00:09:59.099910 7830 consensus_queue.cc:237] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } }
I20260924 00:09:59.100159 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.100068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:60309)
0924 00:09:59.100105 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.100132 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 00:09:59.100613 7286 catalog_manager.cc:5495] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3230a4ecd5214bf7b0cd983f73d35dbf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3230a4ecd5214bf7b0cd983f73d35dbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 } health_report { overall_health: UNKNOWN } } }
I20260924 00:09:59.115499 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.113163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.98.2:34495)
0924 00:09:59.113228 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:09:59.113230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:09:59.113239 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:09:59.113279 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:09:59.113280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:09:59.113303 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:09:59.113362 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:09:59.113498 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.113945 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.114714 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:09:59.114888 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:09:59.114937 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:09:59.115087 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:09:59.115184 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:09:59.115320 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:09:59.115375 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 00:09:59.115393 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:09:59.116590 7692 raft_consensus.cc:1261] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Refusing update from remote peer 3230a4ecd5214bf7b0cd983f73d35dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:59.116622 7419 raft_consensus.cc:1261] T 2f9a7bdd5dd34fc7a090f7908969a706 P 25c93a21847b4defa92167f368274440 [term 1 FOLLOWER]: Refusing update from remote peer 3230a4ecd5214bf7b0cd983f73d35dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:09:59.116935 7830 consensus_queue.cc:1035] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ea89f0b12c40f09fae826ead7b8bbc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:59.117110 7836 consensus_queue.cc:1035] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "25c93a21847b4defa92167f368274440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:09:59.118134 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.118021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:34705)
0924 00:09:59.118064 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.118095 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 00:09:59.118134 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.118022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:48175)
0924 00:09:59.118066 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.118096 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 00:09:59.119089 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.118996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.3:47917)
0924 00:09:59.119033 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.119058 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:09:59.119145 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.119057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:41565)
0924 00:09:59.119094 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.119116 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:09:59.119444 7839 mvcc.cc:205] Tried to move back new op lower bound from 7332694421979328512 to 7332694421913915392. Current Snapshot: MvccSnapshot[applied={T|T < 7332694421979328512}]
I20260924 00:09:59.119719 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.119629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.2:36777)
0924 00:09:59.119670 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.119694 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 00:09:59.120232 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.120143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45797 (local address 127.0.98.1:58301)
0924 00:09:59.120181 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.120206 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 00:09:59.120352 7841 mvcc.cc:205] Tried to move back new op lower bound from 7332694421979328512 to 7332694421913915392. Current Snapshot: MvccSnapshot[applied={T|T < 7332694421979328512}]
I20260924 00:09:59.122084 7284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:59.122179 7284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:09:59.123471 7284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc409f33c33c4e2a84d324fc033beb19: Sending DeleteTablet for 3 replicas of tablet 2f9a7bdd5dd34fc7a090f7908969a706
I20260924 00:09:59.123816 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41530
I20260924 00:09:59.123858 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56262
I20260924 00:09:59.123827 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a7bdd5dd34fc7a090f7908969a706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:09:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35946
I20260924 00:09:59.124361 7849 tablet_replica.cc:330] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf: stopping tablet replica
I20260924 00:09:59.124425 7849 raft_consensus.cc:2229] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:09:59.124481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7247
I20260924 00:09:59.124500 7849 raft_consensus.cc:2258] T 2f9a7bdd5dd34fc7a090f7908969a706 P 3230a4ecd5214bf7b0cd983f73d35dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:59.125033 7850 tablet_replica.cc:330] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: stopping tablet replica
I20260924 00:09:59.125097 7850 raft_consensus.cc:2229] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:09:59.125147 7850 raft_consensus.cc:2258] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:09:59.126621 7850 ts_tablet_manager.cc:1905] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:09:59.128523 7850 ts_tablet_manager.cc:1918] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:09:59.128587 7850 log.cc:1196] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/2f9a7bdd5dd34fc7a090f7908969a706
I20260924 00:09:59.128815 7850 ts_tablet_manager.cc:1939] T 2f9a7bdd5dd34fc7a090f7908969a706 P 54ea89f0b12c40f09fae826ead7b8bbc: Deleting consensus metadata
I20260924 00:09:59.129179 7273 catalog_manager.cc:4841] TS 54ea89f0b12c40f09fae826ead7b8bbc (127.0.98.3:36029): tablet 2f9a7bdd5dd34fc7a090f7908969a706 (table test-table [id=4975d0411efb4017ae8689c40d564835]) successfully deleted
I20260924 00:09:59.132833 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.132690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45075 (local address 127.0.0.1:41540)
0924 00:09:59.132757 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.132789 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:09:59.132997 7273 catalog_manager.cc:4584] TS 25c93a21847b4defa92167f368274440 (127.0.98.1:45075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f9a7bdd5dd34fc7a090f7908969a706: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45075: connect: Connection refused (error 111)
I20260924 00:09:59.133034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7473
I20260924 00:09:59.138322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7609
I20260924 00:09:59.138477 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:09:59.138292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34495 (local address 127.0.0.1:56270)
0924 00:09:59.138371 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 00:09:59.138409 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 00:09:59.143708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 6966
I20260924 00:09:59.148998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7154
2026-09-24T00:09:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4872 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:09:59 dist-test-slave-3q8k krb5kdc[7853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:09:59 dist-test-slave-3q8k krb5kdc[7853](info): set up 2 sockets
Sep 24 00:09:59 dist-test-slave-3q8k krb5kdc[7853](info): commencing operation
krb5kdc: starting...
W20260924 00:10:00.219851 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 00:10:00.338843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:00.421603 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:00.421885 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:00.421926 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:00.423182 7866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:00.423290 7866 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:10:00.423316 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:00.423347 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:00.423377 7866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:00.423408 7866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:00.424968 7866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34097
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:00.425275 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:00.425527 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:00.430478 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.430648 7866 server_base.cc:1034] running on GCE node
W20260924 00:10:00.430505 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.430507 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.431052 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:00.431404 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:00.432591 7866 hybrid_clock.cc:648] HybridClock initialized: now 1790208600432593 us; error 56 us; skew 500 ppm
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:00.435001 7866 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 00:10:00.435523 7866 webserver.cc:460] Webserver started at http://127.0.98.62:46661/ using document root <none> and password file <none>
I20260924 00:10:00.435778 7866 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:00.435853 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:00.435999 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:00.437063 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "f5b6a5840ce2400d930eaecb91073433"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "a15a2c9e3209b935862188b92f91069c"
server_key_iv: "73ad4d060b5c4cad308e3fa80cd0f781"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.437448 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5b6a5840ce2400d930eaecb91073433"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "a15a2c9e3209b935862188b92f91069c"
server_key_iv: "73ad4d060b5c4cad308e3fa80cd0f781"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.438804 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:10:00.439594 7881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.439760 7866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.439873 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "f5b6a5840ce2400d930eaecb91073433"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "a15a2c9e3209b935862188b92f91069c"
server_key_iv: "73ad4d060b5c4cad308e3fa80cd0f781"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.439951 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:00.466377 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:00.468504 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:00.468701 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:00.473783 7866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34097
I20260924 00:10:00.473829 7943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34097 every 8 connection(s)
I20260924 00:10:00.474171 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:00.474797 7944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:00.477072 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Bootstrap starting.
I20260924 00:10:00.477658 7944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:00.477953 7944 log.cc:824] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:00.478618 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: No bootstrap required, opened a new log
I20260924 00:10:00.480816 7944 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260924 00:10:00.480962 7944 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:00.481020 7944 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b6a5840ce2400d930eaecb91073433, State: Initialized, Role: FOLLOWER
I20260924 00:10:00.481178 7944 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260924 00:10:00.481240 7944 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:00.481262 7944 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:00.481334 7944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:00.481860 7944 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260924 00:10:00.481984 7944 leader_election.cc:302] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b6a5840ce2400d930eaecb91073433; no voters:
I20260924 00:10:00.482175 7944 leader_election.cc:288] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:10:00.482260 7947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:00.482422 7947 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [term 1 LEADER]: Becoming Leader. State: Replica: f5b6a5840ce2400d930eaecb91073433, State: Running, Role: LEADER
I20260924 00:10:00.482445 7944 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:00.482519 7947 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } }
I20260924 00:10:00.482909 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5b6a5840ce2400d930eaecb91073433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260924 00:10:00.483088 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:00.482954 7949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b6a5840ce2400d930eaecb91073433. Latest consensus state: current_term: 1 leader_uuid: "f5b6a5840ce2400d930eaecb91073433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6a5840ce2400d930eaecb91073433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34097 } } }
I20260924 00:10:00.483207 7949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:00.483556 7956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:00.484000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7866
I20260924 00:10:00.484091 7956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:00.484103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:10:00.484297 392 external_mini_cluster.cc:1428] Setting key 8b7006b41823931fac0ba29305bb2cb6
I20260924 00:10:00.485658 7956 catalog_manager.cc:1347] Generated new cluster ID: fc0c3139a30349f8ad67b93f03b858be
I20260924 00:10:00.485713 7956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:00.495082 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.484816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.98.62:34097)
0924 00:10:00.485191 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.485201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.485408 (+ 207us) server_negotiation.cc:407] Connection header received
0924 00:10:00.485604 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.485612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.485673 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.485814 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.486526 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.487628 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.488383 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.488867 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.489075 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.491600 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.491609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.491609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.491621 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.493385 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.493788 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.493792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.493792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.493891 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.494118 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.494119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.494119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.494373 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.494545 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.494631 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.494889 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":65,"threads_started":1}
I20260924 00:10:00.503343 7956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:10:00.504142 7956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:00.511530 7956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Generated new TSK 0
I20260924 00:10:00.511776 7956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 00:10:00.538385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:00.621398 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:00.621681 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:00.621726 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:00.623442 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:00.623526 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:00.623597 7970 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:10:00.623641 7970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:00.625932 7970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:00.626245 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:00.626502 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:00.631953 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.631956 7978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.631958 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.632166 7970 server_base.cc:1034] running on GCE node
I20260924 00:10:00.632452 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:00.632772 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:00.633932 7970 hybrid_clock.cc:648] HybridClock initialized: now 1790208600633901 us; error 51 us; skew 500 ppm
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:00.636794 7970 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:10:00.637248 7970 webserver.cc:460] Webserver started at http://127.0.98.1:40747/ using document root <none> and password file <none>
I20260924 00:10:00.637497 7970 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:00.637568 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:00.637714 7970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:00.638783 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a697c8b942c41268fda250d5274f608"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "f461a98dddf1c9a84399dbc416ae28bb"
server_key_iv: "3f1084651de57018a1d088d09980b2d6"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.639161 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a697c8b942c41268fda250d5274f608"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "f461a98dddf1c9a84399dbc416ae28bb"
server_key_iv: "3f1084651de57018a1d088d09980b2d6"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.640578 7970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:00.641316 7985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.641534 7970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:00.641635 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "0a697c8b942c41268fda250d5274f608"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "f461a98dddf1c9a84399dbc416ae28bb"
server_key_iv: "3f1084651de57018a1d088d09980b2d6"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.641716 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:00.652073 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:00.654244 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:00.654417 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:00.655037 7970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:00.655107 7970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.655153 7970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:00.655184 7970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.661916 7970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46189
I20260924 00:10:00.662029 8100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46189 every 8 connection(s)
I20260924 00:10:00.662328 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:10:00.663384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 7970
I20260924 00:10:00.663471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:10:00.663631 392 external_mini_cluster.cc:1428] Setting key de4b83a7f7dbe38269b3f1ee3c840291
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:00.669963 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.655624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58109 (local address 127.0.98.62:34097)
0924 00:10:00.655704 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.655707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.655927 (+ 220us) server_negotiation.cc:407] Connection header received
0924 00:10:00.656436 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.656437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.656450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.656541 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.657401 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.658242 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.659881 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.660226 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.660316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.666774 (+ 6458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.666776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.666776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.666784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.668380 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.668915 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.668916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.668917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.668978 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.669269 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.669270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.669270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.669400 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.669482 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.669771 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.669863 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:10:00.670167 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.655613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:58109)
0924 00:10:00.655790 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.655831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.656338 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.656609 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.656613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.656997 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.657311 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.657318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.658339 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.658340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.659757 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.659761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.660324 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.660325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.660430 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.664995 (+ 4565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.665006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.666648 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.668513 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.668521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.668524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.668804 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.669067 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.669068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.669068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.669188 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.669470 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.669477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.669633 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.669747 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.669927 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260924 00:10:00.670781 8101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260924 00:10:00.670888 8101 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:00.671103 8101 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:10:00.671762 7897 ts_manager.cc:194] Registered new tserver with Master: 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189)
I20260924 00:10:00.672694 7897 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 00:10:00.677920 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.671975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54101 (local address 127.0.98.62:34097)
0924 00:10:00.672027 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.672029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.672157 (+ 128us) server_negotiation.cc:407] Connection header received
0924 00:10:00.672224 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.672225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.672233 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.672278 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.672603 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.673204 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.674154 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.674329 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.674383 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.675448 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.675450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.675450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.675458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.676850 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.677248 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.677248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.677249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.677294 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.677496 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.677496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.677496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.677591 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.677648 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.677803 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.677860 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:10:00.678035 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.671908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.1:54101)
0924 00:10:00.672093 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.672105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.672170 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.672284 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.672284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.672475 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.672545 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.672546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.673284 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.673285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.674082 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.674084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.674401 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.674402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.674468 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.674958 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.674963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.675368 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.676938 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.676939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.676939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.677174 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.677357 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.677358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.677358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.677435 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.677652 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.677653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.677712 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.677788 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.677901 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":27,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 00:10:00.690752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:00.773541 8110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:00.773788 8110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:00.773813 8110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:00.774931 8110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:00.774972 8110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:00.774992 8110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:10:00.775010 8110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:00.776561 8110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:00.776762 8110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:00.776980 8110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:00.782043 8118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.782004 8116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.782028 8115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.782280 8110 server_base.cc:1034] running on GCE node
I20260924 00:10:00.782601 8110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:00.782824 8110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:00.783967 8110 hybrid_clock.cc:648] HybridClock initialized: now 1790208600783947 us; error 39 us; skew 500 ppm
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:00.786567 8110 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:10:00.787092 8110 webserver.cc:460] Webserver started at http://127.0.98.2:38983/ using document root <none> and password file <none>
I20260924 00:10:00.787385 8110 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:00.787464 8110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:00.787622 8110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:00.788702 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b3e2ae4412c4273ab84f7542e80939d"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "6d2766136577516b2b6eecfc0cdea8eb"
server_key_iv: "2a60d080addfc97f1d4ddd7225547be5"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.789090 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b3e2ae4412c4273ab84f7542e80939d"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "6d2766136577516b2b6eecfc0cdea8eb"
server_key_iv: "2a60d080addfc97f1d4ddd7225547be5"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.790534 8110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:10:00.791314 8125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.791492 8110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:00.791600 8110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "3b3e2ae4412c4273ab84f7542e80939d"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "6d2766136577516b2b6eecfc0cdea8eb"
server_key_iv: "2a60d080addfc97f1d4ddd7225547be5"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.791685 8110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:00.805631 8110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:00.808127 8110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:00.808303 8110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:00.808899 8110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:00.808964 8110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.808996 8110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:00.809036 8110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.816177 8110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39509
I20260924 00:10:00.816323 8240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39509 every 8 connection(s)
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:00.816643 8110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:10:00.819808 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.809335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54591 (local address 127.0.98.62:34097)
0924 00:10:00.809406 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.809408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.809644 (+ 236us) server_negotiation.cc:407] Connection header received
0924 00:10:00.810176 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.810177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.810189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.810265 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.811274 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.812104 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.813817 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.814029 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.814124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.817098 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.817100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.817100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.817108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.818535 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.818918 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.818919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.818919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.818969 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.819185 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.819186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.819186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.819348 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.819402 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.819681 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.819743 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:10:00.819921 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.809327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:54591)
0924 00:10:00.809493 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.809534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.810060 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.810351 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.810355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.810790 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.811106 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.811113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.812214 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.812215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.813695 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.813698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.814132 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.814133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.814291 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.814985 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.814996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.816979 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.818626 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.818631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.818632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.818846 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.819031 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.819031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.819031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.819128 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.819408 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.819412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.819531 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.819617 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.819734 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260924 00:10:00.820398 8241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260924 00:10:00.820482 8241 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:00.820667 8241 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260924 00:10:00.821111 7898 ts_manager.cc:194] Registered new tserver with Master: 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509)
I20260924 00:10:00.821611 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54591
I20260924 00:10:00.826056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8110
I20260924 00:10:00.826156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:10:00.826383 392 external_mini_cluster.cc:1428] Setting key 470d4c394f5d7b410144c6d626f482c1
I20260924 00:10:00.827555 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.821512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54225 (local address 127.0.98.62:34097)
0924 00:10:00.821561 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.821563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.821706 (+ 143us) server_negotiation.cc:407] Connection header received
0924 00:10:00.821752 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.821753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.821764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.821820 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.822181 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.822732 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.823469 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.823709 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.823763 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.824610 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.824611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.824611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.824617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.826096 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.826553 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.826555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.826555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.826617 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.826843 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.826844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.826844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.827076 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.827146 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.827425 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.827492 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:10:00.827603 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.821442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.2:54225)
0924 00:10:00.821616 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.821625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.821697 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.821835 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.821836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.822034 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.822102 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.822104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.822814 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.822815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.823400 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.823402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.823775 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.823776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.823826 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.824223 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.824227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.824516 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.826232 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.826233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.826233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.826406 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.826714 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.826715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.826715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.826775 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.827156 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.827157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.827210 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.827311 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.827427 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:10:00.853927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34097
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:00.941371 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:00.941623 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:00.941668 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:00.942888 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:00.942953 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:00.942991 8250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:10:00.943048 8250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:00.944650 8250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:00.944953 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:00.945201 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:00.950280 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.950315 8250 server_base.cc:1034] running on GCE node
W20260924 00:10:00.950239 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:00.950421 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:00.950649 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:00.950905 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:00.952068 8250 hybrid_clock.cc:648] HybridClock initialized: now 1790208600952055 us; error 25 us; skew 500 ppm
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:00.954643 8250 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:10:00.955065 8250 webserver.cc:460] Webserver started at http://127.0.98.3:33407/ using document root <none> and password file <none>
I20260924 00:10:00.955363 8250 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:00.955436 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:00.955595 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:00.956662 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "14f9c688fffd477fae58e802f5367e93"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "44b84cc83e6a3298c955a407a37e81e1"
server_key_iv: "0eab6a63f15ccf9f726c2f6ab339ce6d"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.957041 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14f9c688fffd477fae58e802f5367e93"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "44b84cc83e6a3298c955a407a37e81e1"
server_key_iv: "0eab6a63f15ccf9f726c2f6ab339ce6d"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.958356 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:00.959121 8265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.959303 8250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:00.959405 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "14f9c688fffd477fae58e802f5367e93"
format_stamp: "Formatted at 2026-09-24 00:10:00 on dist-test-slave-3q8k"
server_key: "44b84cc83e6a3298c955a407a37e81e1"
server_key_iv: "0eab6a63f15ccf9f726c2f6ab339ce6d"
server_key_version: "encryptionkey@0"
I20260924 00:10:00.959523 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:00.970841 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:00.973243 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:00.973410 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:00.973904 8250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:00.973973 8250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:00.974009 8250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:00.974038 8250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:00.981274 8250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43909
I20260924 00:10:00.981355 8380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43909 every 8 connection(s)
I20260924 00:10:00.981680 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:10:00.984815 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.974396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45873 (local address 127.0.98.62:34097)
0924 00:10:00.974466 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.974469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.974693 (+ 224us) server_negotiation.cc:407] Connection header received
0924 00:10:00.975273 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.975275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.975290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.975374 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.976019 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.976838 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.978502 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.978874 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.978946 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.981680 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.981681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.981681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.981688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.983356 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.983860 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.983862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.983862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.983919 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.984179 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.984180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.984181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.984301 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.984380 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.984647 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.984722 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:10:00.985024 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.974369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:45873)
0924 00:10:00.974541 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.974597 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.975108 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.975463 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.975467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.975717 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.975904 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.975932 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.976943 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.976944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.978374 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.978377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.978970 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.978971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.979125 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.979708 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.979716 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.981588 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.983472 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.983478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.983480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.983784 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.983998 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.983999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.983999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.984110 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.984371 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.984377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.984515 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.984626 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.984779 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260924 00:10:00.985476 8381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34097
I20260924 00:10:00.985561 8381 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:00.985742 8381 heartbeater.cc:511] Master 127.0.98.62:34097 requested a full tablet report, sending...
I20260924 00:10:00.986182 7897 ts_manager.cc:194] Registered new tserver with Master: 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:00.986676 7897 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45873
I20260924 00:10:00.989266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8250
I20260924 00:10:00.989384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:10:00.989593 392 external_mini_cluster.cc:1428] Setting key 6e9266e2144018b2e37f8e2d8954abcb
I20260924 00:10:00.990471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:10:00.992410 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.986557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52753 (local address 127.0.98.62:34097)
0924 00:10:00.986598 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:00.986600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:00.986647 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:10:00.986714 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:00.986715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:00.986722 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:00.986797 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:00.987254 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.987721 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.988434 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.988645 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.988718 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.989790 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:00.989793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:00.989793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:00.989802 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:00.991273 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.991664 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.991665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.991665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.991713 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:00.991908 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:00.991908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:00.991909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:00.992000 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:00.992048 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:00.992201 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:10:00.992261 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:10:00.992452 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:00.986465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.98.3:52753)
0924 00:10:00.986593 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:00.986603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:00.986660 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:00.986874 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:00.986876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:00.987077 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:00.987139 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.987140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.987815 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.987815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.988333 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:00.988335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:00.988716 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:00.988717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:00.988786 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:00.989217 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:00.989224 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:00.989680 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:00.991352 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.991353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.991354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.991591 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.991775 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:00.991776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:00.991777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:00.991852 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:00.992061 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:00.992061 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:00.992106 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 00:10:00.992185 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:10:00.992306 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
Sep 24 00:10:00 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:10:01.012544 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:01.006800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.62:34097)
0924 00:10:01.006869 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.006871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.006880 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:01.006911 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.006911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.006920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.006971 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:01.007673 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.008173 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.008930 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.009129 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.009191 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.012418 (+ 3227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57734: BlockingRecv error: recv got EOF from 127.0.0.1:57734 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:01.019675 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.013266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.0.98.62:34097)
0924 00:10:01.013322 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.013323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.013375 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:10:01.013435 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.013435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.013442 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.013496 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:01.013940 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.014401 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.015042 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.015214 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.015320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.017312 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:01.017314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:01.017314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:01.017320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:01.018733 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.019032 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.019032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.019033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.019081 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.019284 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.019285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.019285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.019389 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:01.019464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.019552 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.019610 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:10:01.021447 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:10:01.023104 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:01.030467 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46189 (local address 127.0.0.1:46908)
0924 00:10:01.025784 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.025792 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.025829 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.026151 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.026152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.026154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.026221 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.026224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027635 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.027636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.029100 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.029102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030112 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.030218 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.030236 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.030299 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.030363 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":212,"thread_start_us":91,"threads_started":1}
I20260924 00:10:01.030467 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43909 (local address 127.0.0.1:38062)
0924 00:10:01.025570 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.025611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.025680 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.026095 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.026100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.026103 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.026184 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.026187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027545 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.027546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.029013 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.029015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030110 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.030219 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.030237 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.030287 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.030363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":122,"thread_start_us":53,"threads_started":1}
I20260924 00:10:01.030817 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39509 (local address 127.0.0.1:34986)
0924 00:10:01.026344 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.026418 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.026469 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.026708 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.026709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.026711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.026743 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.026745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027920 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.027921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.029187 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.029190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030418 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.030500 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.030505 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.030561 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.030602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":64,"threads_started":1}
I20260924 00:10:01.030833 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.0.98.2:39509)
0924 00:10:01.026296 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.026299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.026535 (+ 236us) server_negotiation.cc:407] Connection header received
0924 00:10:01.026575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.026577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.026591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.026653 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.026812 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027840 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.029287 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030216 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030296 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.030378 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.030684 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.030722 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":87,"threads_started":1}
I20260924 00:10:01.031091 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.98.3:43909)
0924 00:10:01.025897 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.025902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.025912 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:10:01.025953 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.025955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.025971 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.026060 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.026278 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027444 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.029193 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030004 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030515 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.030784 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.030810 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.030990 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":85,"threads_started":1}
I20260924 00:10:01.031256 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.025824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.98.1:46189)
0924 00:10:01.025994 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.025999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.026008 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:01.026042 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.026044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.026057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.026357 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.026417 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.027530 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.029186 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.030004 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.030381 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.031038 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.031066 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.031110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260924 00:10:01.031524 8175 tablet_service.cc:1467] Processing CreateTablet for tablet db77640b3e8e4753a7bed31fdf749a1a (DEFAULT_TABLE table=test-table [id=58cd6274273c40e5821f21c946827602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.031742 8315 tablet_service.cc:1467] Processing CreateTablet for tablet db77640b3e8e4753a7bed31fdf749a1a (DEFAULT_TABLE table=test-table [id=58cd6274273c40e5821f21c946827602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.031824 8035 tablet_service.cc:1467] Processing CreateTablet for tablet db77640b3e8e4753a7bed31fdf749a1a (DEFAULT_TABLE table=test-table [id=58cd6274273c40e5821f21c946827602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.031947 8175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db77640b3e8e4753a7bed31fdf749a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.032089 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db77640b3e8e4753a7bed31fdf749a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.032089 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db77640b3e8e4753a7bed31fdf749a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.035166 8409 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Bootstrap starting.
I20260924 00:10:01.035696 8410 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Bootstrap starting.
I20260924 00:10:01.036108 8409 tablet_bootstrap.cc:654] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.036113 8411 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Bootstrap starting.
I20260924 00:10:01.036345 8410 tablet_bootstrap.cc:654] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.036459 8409 log.cc:824] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:01.036681 8410 log.cc:824] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:01.036734 8411 tablet_bootstrap.cc:654] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.037061 8411 log.cc:824] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:01.037410 8409 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: No bootstrap required, opened a new log
I20260924 00:10:01.037523 8409 ts_tablet_manager.cc:1397] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:10:01.037907 8410 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: No bootstrap required, opened a new log
I20260924 00:10:01.037979 8410 ts_tablet_manager.cc:1397] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:10:01.039592 8411 tablet_bootstrap.cc:492] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: No bootstrap required, opened a new log
I20260924 00:10:01.039650 8411 ts_tablet_manager.cc:1397] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:10:01.039613 8410 raft_consensus.cc:348] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.039582 8409 raft_consensus.cc:348] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.039727 8410 raft_consensus.cc:374] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.039748 8410 raft_consensus.cc:729] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.039744 8409 raft_consensus.cc:374] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.039780 8409 raft_consensus.cc:729] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.039873 8410 consensus_queue.cc:260] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.039886 8409 consensus_queue.cc:260] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.040149 8409 ts_tablet_manager.cc:1428] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:10:01.040138 8410 ts_tablet_manager.cc:1428] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:10:01.040153 8381 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260924 00:10:01.041476 8411 raft_consensus.cc:348] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.041584 8411 raft_consensus.cc:374] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.041601 8411 raft_consensus.cc:729] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.041674 8411 consensus_queue.cc:260] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.041833 8411 ts_tablet_manager.cc:1428] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:10:01.041903 8101 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260924 00:10:01.041998 8241 heartbeater.cc:503] Master 127.0.98.62:34097 was elected leader, sending a full tablet report...
I20260924 00:10:01.053194 8416 raft_consensus.cc:482] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:01.053329 8416 raft_consensus.cc:504] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.053681 8416 leader_election.cc:288] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909), 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189)
I20260924 00:10:01.056514 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.053900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56857 (local address 127.0.98.1:46189)
0924 00:10:01.053949 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.053951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.053961 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:10:01.054021 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.054022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.054033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.054123 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.054270 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.054735 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.055778 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.056147 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.056232 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.056242 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.056412 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.056446 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:10:01.056559 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.053882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48071 (local address 127.0.98.3:43909)
0924 00:10:01.053969 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.053971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.054095 (+ 124us) server_negotiation.cc:407] Connection header received
0924 00:10:01.054172 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.054173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.054182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.054238 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.054382 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.054847 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.055839 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.056207 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.056300 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.056310 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.056459 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.056494 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:10:01.056596 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.053828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46189 (local address 127.0.98.2:56857)
0924 00:10:01.053892 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.053906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.053953 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.054127 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.054128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.054130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.054195 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.054196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.054829 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.054829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.055690 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.055692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.056241 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.056242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.056327 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.056330 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.056403 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.056443 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":3}
I20260924 00:10:01.056587 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.053829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43909 (local address 127.0.98.2:48071)
0924 00:10:01.054022 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.054031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.054120 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.054242 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.054243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.054244 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.054311 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.054312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.054931 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.054932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.055751 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.055753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.056304 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.056305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.056371 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.056375 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.056452 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.056494 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":46,"thread_start_us":92,"threads_started":1}
I20260924 00:10:01.056926 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db77640b3e8e4753a7bed31fdf749a1a" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93" is_pre_election: true
I20260924 00:10:01.056916 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db77640b3e8e4753a7bed31fdf749a1a" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608" is_pre_election: true
I20260924 00:10:01.057062 8335 raft_consensus.cc:2454] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 0.
I20260924 00:10:01.057062 8043 raft_consensus.cc:2454] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 0.
I20260924 00:10:01.057297 8129 leader_election.cc:302] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f9c688fffd477fae58e802f5367e93, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.057477 8416 raft_consensus.cc:2781] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:01.057556 8416 raft_consensus.cc:482] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:01.057600 8416 raft_consensus.cc:3037] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.058619 8416 raft_consensus.cc:504] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.058763 8416 leader_election.cc:288] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 election: Requested vote from peers 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909), 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189)
I20260924 00:10:01.058892 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db77640b3e8e4753a7bed31fdf749a1a" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93"
I20260924 00:10:01.058899 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db77640b3e8e4753a7bed31fdf749a1a" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608"
I20260924 00:10:01.058965 8335 raft_consensus.cc:3037] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.058990 8043 raft_consensus.cc:3037] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.059713 8043 raft_consensus.cc:2454] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 1.
I20260924 00:10:01.059719 8335 raft_consensus.cc:2454] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 1.
I20260924 00:10:01.059894 8126 leader_election.cc:302] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.060019 8416 raft_consensus.cc:2781] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:01.060148 8416 raft_consensus.cc:686] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 1 LEADER]: Becoming Leader. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Running, Role: LEADER
I20260924 00:10:01.060246 8416 consensus_queue.cc:237] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } }
I20260924 00:10:01.061038 7898 catalog_manager.cc:5495] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3b3e2ae4412c4273ab84f7542e80939d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } health_report { overall_health: HEALTHY } } }
W20260924 00:10:01.067283 8242 tablet.cc:2369] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:10:01.070271 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.067922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.98.2:39509)
0924 00:10:01.067986 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.067988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.067997 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:01.068031 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.068032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.068050 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.068134 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.068247 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.068704 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.069470 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.069660 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.069735 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.069932 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.070043 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.070133 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.070175 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.070193 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:10:01.071998 8043 raft_consensus.cc:1261] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Refusing update from remote peer 3b3e2ae4412c4273ab84f7542e80939d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.072007 8335 raft_consensus.cc:1261] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Refusing update from remote peer 3b3e2ae4412c4273ab84f7542e80939d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.072321 8416 consensus_queue.cc:1035] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.072465 8419 consensus_queue.cc:1035] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.074682 8422 mvcc.cc:205] Tried to move back new op lower bound from 7332694429988495360 to 7332694429943300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694429988495360}]
I20260924 00:10:01.074921 8424 mvcc.cc:205] Tried to move back new op lower bound from 7332694429988495360 to 7332694429943300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694429988495360}]
I20260924 00:10:01.075179 8423 mvcc.cc:205] Tried to move back new op lower bound from 7332694429988495360 to 7332694429943300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694429988495360}]
I20260924 00:10:01.077245 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.077346 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.078083 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Sending DeleteTablet for 3 replicas of tablet db77640b3e8e4753a7bed31fdf749a1a
I20260924 00:10:01.078385 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet db77640b3e8e4753a7bed31fdf749a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38062
I20260924 00:10:01.078649 8432 tablet_replica.cc:330] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: stopping tablet replica
I20260924 00:10:01.078728 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet db77640b3e8e4753a7bed31fdf749a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260924 00:10:01.078750 8175 tablet_service.cc:1514] Processing DeleteTablet for tablet db77640b3e8e4753a7bed31fdf749a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260924 00:10:01.078796 8432 raft_consensus.cc:2229] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.078940 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:10:01.079094 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:01.079298 8432 raft_consensus.cc:2258] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.079422 8433 tablet_replica.cc:330] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: stopping tablet replica
I20260924 00:10:01.079523 8433 raft_consensus.cc:2229] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:10:01.079916 8432 ts_tablet_manager.cc:1905] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.080415 8035 tablet_service.cc:1467] Processing CreateTablet for tablet f4571829b0b945f99721a73154c88043 (DEFAULT_TABLE table=test-table [id=ba0d49e9dc9b46d6af74622c9388594f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.080502 8175 tablet_service.cc:1467] Processing CreateTablet for tablet f4571829b0b945f99721a73154c88043 (DEFAULT_TABLE table=test-table [id=ba0d49e9dc9b46d6af74622c9388594f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.080539 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4571829b0b945f99721a73154c88043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.080608 8175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4571829b0b945f99721a73154c88043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.081907 8409 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Bootstrap starting.
I20260924 00:10:01.081971 8315 tablet_service.cc:1467] Processing CreateTablet for tablet f4571829b0b945f99721a73154c88043 (DEFAULT_TABLE table=test-table [id=ba0d49e9dc9b46d6af74622c9388594f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.082154 8411 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Bootstrap starting.
I20260924 00:10:01.082274 8433 raft_consensus.cc:2258] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.082352 8409 tablet_bootstrap.cc:654] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.082453 8434 tablet_replica.cc:330] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: stopping tablet replica
I20260924 00:10:01.082561 8434 raft_consensus.cc:2229] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.082615 8411 tablet_bootstrap.cc:654] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.082657 8434 raft_consensus.cc:2258] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.082834 8432 ts_tablet_manager.cc:1918] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.082857 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4571829b0b945f99721a73154c88043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.082929 8432 log.cc:1196] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/db77640b3e8e4753a7bed31fdf749a1a
I20260924 00:10:01.083190 8432 ts_tablet_manager.cc:1939] T db77640b3e8e4753a7bed31fdf749a1a P 14f9c688fffd477fae58e802f5367e93: Deleting consensus metadata
I20260924 00:10:01.083593 7885 catalog_manager.cc:4841] TS 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909): tablet db77640b3e8e4753a7bed31fdf749a1a (table test-table [id=58cd6274273c40e5821f21c946827602]) successfully deleted
I20260924 00:10:01.084095 8411 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: No bootstrap required, opened a new log
I20260924 00:10:01.084152 8411 ts_tablet_manager.cc:1397] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:10:01.084153 8410 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Bootstrap starting.
I20260924 00:10:01.084098 8434 ts_tablet_manager.cc:1905] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.084342 8433 ts_tablet_manager.cc:1905] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.084311 8411 raft_consensus.cc:348] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.084386 8411 raft_consensus.cc:374] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.084406 8411 raft_consensus.cc:729] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.084453 8411 consensus_queue.cc:260] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.084537 8410 tablet_bootstrap.cc:654] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.084553 8411 ts_tablet_manager.cc:1428] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.085819 8433 ts_tablet_manager.cc:1918] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.085887 8433 log.cc:1196] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/db77640b3e8e4753a7bed31fdf749a1a
I20260924 00:10:01.085969 8409 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: No bootstrap required, opened a new log
I20260924 00:10:01.086014 8409 ts_tablet_manager.cc:1397] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:10:01.086073 8433 ts_tablet_manager.cc:1939] T db77640b3e8e4753a7bed31fdf749a1a P 3b3e2ae4412c4273ab84f7542e80939d: Deleting consensus metadata
I20260924 00:10:01.086155 8409 raft_consensus.cc:348] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.086220 8409 raft_consensus.cc:374] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.086244 8434 ts_tablet_manager.cc:1918] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.086241 8409 raft_consensus.cc:729] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.086313 8434 log.cc:1196] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/db77640b3e8e4753a7bed31fdf749a1a
I20260924 00:10:01.086310 8409 consensus_queue.cc:260] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.086397 8409 ts_tablet_manager.cc:1428] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.086838 8434 ts_tablet_manager.cc:1939] T db77640b3e8e4753a7bed31fdf749a1a P 0a697c8b942c41268fda250d5274f608: Deleting consensus metadata
I20260924 00:10:01.086882 7882 catalog_manager.cc:4841] TS 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509): tablet db77640b3e8e4753a7bed31fdf749a1a (table test-table [id=58cd6274273c40e5821f21c946827602]) successfully deleted
I20260924 00:10:01.087270 7882 catalog_manager.cc:4841] TS 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189): tablet db77640b3e8e4753a7bed31fdf749a1a (table test-table [id=58cd6274273c40e5821f21c946827602]) successfully deleted
I20260924 00:10:01.087634 8410 tablet_bootstrap.cc:492] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: No bootstrap required, opened a new log
I20260924 00:10:01.087698 8410 ts_tablet_manager.cc:1397] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:10:01.087865 8410 raft_consensus.cc:348] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.087931 8410 raft_consensus.cc:374] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.087946 8410 raft_consensus.cc:729] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.088007 8410 consensus_queue.cc:260] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.088099 8410 ts_tablet_manager.cc:1428] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.124119 8417 raft_consensus.cc:482] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:01.124222 8417 raft_consensus.cc:504] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.124631 8417 leader_election.cc:288] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.128284 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.124794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39509 (local address 127.0.98.1:33279)
0924 00:10:01.124856 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.124868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.124911 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.125197 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.125198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.125200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.125295 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.125297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.126243 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.126244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.127371 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.127374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.128021 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.128022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.128096 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.128100 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.128149 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.128195 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260924 00:10:01.128511 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.124943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50815 (local address 127.0.98.3:43909)
0924 00:10:01.124990 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.124992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.125144 (+ 152us) server_negotiation.cc:407] Connection header received
0924 00:10:01.125191 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.125192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.125202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.125255 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.125460 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.125930 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.127464 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.128065 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.128156 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.128167 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.128384 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.128439 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:10:01.128615 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.124994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33279 (local address 127.0.98.2:39509)
0924 00:10:01.125038 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.125040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.125048 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:01.125081 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.125082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.125092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.125146 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.125390 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.126156 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.127492 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.127917 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.127978 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.127989 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.128497 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.128540 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:10:01.128857 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4571829b0b945f99721a73154c88043" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d" is_pre_election: true
I20260924 00:10:01.128962 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.124818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43909 (local address 127.0.98.1:50815)
0924 00:10:01.125052 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.125064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.125132 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.125271 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.125272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.125274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.125345 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.125347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.126022 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.126023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.127375 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.127377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.128205 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.128206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.128287 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.128291 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.128349 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.128865 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260924 00:10:01.129005 8195 raft_consensus.cc:2454] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 0.
I20260924 00:10:01.129194 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4571829b0b945f99721a73154c88043" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93" is_pre_election: true
I20260924 00:10:01.129207 7986 leader_election.cc:302] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.129284 8335 raft_consensus.cc:2454] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 0.
I20260924 00:10:01.129350 8417 raft_consensus.cc:2781] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:01.129384 8417 raft_consensus.cc:482] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:01.129407 8417 raft_consensus.cc:3037] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.130097 8417 raft_consensus.cc:504] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.130257 8417 leader_election.cc:288] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 election: Requested vote from peers 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.130452 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4571829b0b945f99721a73154c88043" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d"
I20260924 00:10:01.130461 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4571829b0b945f99721a73154c88043" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93"
I20260924 00:10:01.130538 8335 raft_consensus.cc:3037] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.130539 8195 raft_consensus.cc:3037] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.131186 8195 raft_consensus.cc:2454] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 1.
I20260924 00:10:01.131366 8335 raft_consensus.cc:2454] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 1.
I20260924 00:10:01.131385 7986 leader_election.cc:302] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.131492 8417 raft_consensus.cc:2781] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:01.131603 8417 raft_consensus.cc:686] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Running, Role: LEADER
I20260924 00:10:01.131677 8417 consensus_queue.cc:237] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.132438 7898 catalog_manager.cc:5495] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697c8b942c41268fda250d5274f608 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0a697c8b942c41268fda250d5274f608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } health_report { overall_health: UNKNOWN } } }
W20260924 00:10:01.163064 8102 tablet.cc:2369] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:01.167804 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.160048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.0.98.62:34097)
0924 00:10:01.160101 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.160104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.160246 (+ 142us) server_negotiation.cc:407] Connection header received
0924 00:10:01.160318 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.160319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.160327 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.160379 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:01.160997 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.161476 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.162593 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.162785 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.162837 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.165206 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:01.165207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:01.165208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:01.165214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:01.166609 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.166998 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.166999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.167000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.167053 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.167263 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.167264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.167264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.167375 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:01.167450 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.167608 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.167692 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:10:01.167904 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.159940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34097 (local address 127.0.0.1:57738)
0924 00:10:01.160155 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.160169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.160253 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.160410 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.160411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.160838 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:01.160906 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.160910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.161607 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.161608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.162510 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.162513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.162860 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.162862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.162925 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.163603 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:01.163608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:01.165099 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:01.166710 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:01.166718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:01.166720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:01.166909 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:01.167118 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:01.167119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:01.167119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:01.167176 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:01.167441 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:01.167447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:01.167500 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.167584 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.167764 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":79,"thread_start_us":113,"threads_started":1}
I20260924 00:10:01.169872 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:10:01.170083 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:01.171521 8175 tablet_service.cc:1467] Processing CreateTablet for tablet d67244a67e384510b7082d8e60382be2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af88c1722d844fcbbcd99d8b8a4c050b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:10:01.171571 8035 tablet_service.cc:1467] Processing CreateTablet for tablet d67244a67e384510b7082d8e60382be2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af88c1722d844fcbbcd99d8b8a4c050b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:10:01.171610 8315 tablet_service.cc:1467] Processing CreateTablet for tablet d67244a67e384510b7082d8e60382be2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af88c1722d844fcbbcd99d8b8a4c050b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:10:01.171700 8175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67244a67e384510b7082d8e60382be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.171726 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67244a67e384510b7082d8e60382be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.171744 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67244a67e384510b7082d8e60382be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.173221 8410 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: Bootstrap starting.
I20260924 00:10:01.173221 8409 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: Bootstrap starting.
I20260924 00:10:01.173292 8411 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: Bootstrap starting.
I20260924 00:10:01.173691 8410 tablet_bootstrap.cc:654] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.173694 8411 tablet_bootstrap.cc:654] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.173872 8409 tablet_bootstrap.cc:654] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.174448 8410 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: No bootstrap required, opened a new log
I20260924 00:10:01.174500 8410 ts_tablet_manager.cc:1397] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:01.174649 8410 raft_consensus.cc:348] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.174719 8409 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: No bootstrap required, opened a new log
I20260924 00:10:01.174744 8410 raft_consensus.cc:374] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.174758 8409 ts_tablet_manager.cc:1397] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:10:01.174775 8410 raft_consensus.cc:729] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.174822 8410 consensus_queue.cc:260] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.174913 8410 ts_tablet_manager.cc:1428] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:01.174888 8409 raft_consensus.cc:348] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.174957 8409 raft_consensus.cc:374] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.174979 8409 raft_consensus.cc:729] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.175020 8409 consensus_queue.cc:260] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.175100 8411 tablet_bootstrap.cc:492] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: No bootstrap required, opened a new log
I20260924 00:10:01.175107 8409 ts_tablet_manager.cc:1428] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.175060 8415 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.175141 8415 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.175153 8411 ts_tablet_manager.cc:1397] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:10:01.175240 8427 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.175318 8427 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.175344 8411 raft_consensus.cc:348] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.175414 8411 raft_consensus.cc:374] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.175433 8411 raft_consensus.cc:729] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.175473 8411 consensus_queue.cc:260] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.175557 8411 ts_tablet_manager.cc:1428] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:01.175673 8417 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.175730 8417 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 00:10:01.232618 8382 tablet.cc:2369] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:10:01.309546 8415 raft_consensus.cc:482] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:01.309653 8415 raft_consensus.cc:504] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.310021 8415 leader_election.cc:288] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189), 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509)
I20260924 00:10:01.313684 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.310270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46189 (local address 127.0.98.3:48017)
0924 00:10:01.310355 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.310369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.310418 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.310622 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.310623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.310625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.310718 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.310720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.311614 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.311615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.312600 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.312602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.313339 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.313340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.313468 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.313472 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.313559 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.313604 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 00:10:01.313724 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.310339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48017 (local address 127.0.98.1:46189)
0924 00:10:01.310428 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.310430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.310446 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:10:01.310485 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.310486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.310499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.310578 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.310805 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.311529 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.312692 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.313247 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.313322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.313332 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.313582 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.313627 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:10:01.313936 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67244a67e384510b7082d8e60382be2" candidate_uuid: "14f9c688fffd477fae58e802f5367e93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608" is_pre_election: true
I20260924 00:10:01.314040 8043 raft_consensus.cc:2454] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f9c688fffd477fae58e802f5367e93 in term 0.
I20260924 00:10:01.314229 8266 leader_election.cc:302] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 14f9c688fffd477fae58e802f5367e93; no voters:
I20260924 00:10:01.314409 8415 raft_consensus.cc:2781] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:01.314459 8415 raft_consensus.cc:482] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:01.314472 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.310503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59029 (local address 127.0.98.2:39509)
0924 00:10:01.310594 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.310596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.310733 (+ 137us) server_negotiation.cc:407] Connection header received
0924 00:10:01.310770 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.310771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.310785 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.310845 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:01.311160 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.311781 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.313561 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.314150 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.314226 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.314236 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.314354 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.314397 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:10:01.314502 8415 raft_consensus.cc:3037] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.314627 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.310418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39509 (local address 127.0.98.3:59029)
0924 00:10:01.310631 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.310641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.310697 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.310960 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.310961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.310963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:01.311037 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.311038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.311873 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.311873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.313449 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.313452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.314223 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.314224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.314280 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.314283 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.314325 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.314385 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":56,"thread_start_us":140,"threads_started":1}
I20260924 00:10:01.314893 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67244a67e384510b7082d8e60382be2" candidate_uuid: "14f9c688fffd477fae58e802f5367e93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d" is_pre_election: true
I20260924 00:10:01.314982 8195 raft_consensus.cc:2454] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f9c688fffd477fae58e802f5367e93 in term 0.
I20260924 00:10:01.315476 8415 raft_consensus.cc:504] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.315619 8415 leader_election.cc:288] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [CANDIDATE]: Term 1 election: Requested vote from peers 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189), 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509)
I20260924 00:10:01.315786 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67244a67e384510b7082d8e60382be2" candidate_uuid: "14f9c688fffd477fae58e802f5367e93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d"
I20260924 00:10:01.315773 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67244a67e384510b7082d8e60382be2" candidate_uuid: "14f9c688fffd477fae58e802f5367e93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608"
I20260924 00:10:01.315856 8043 raft_consensus.cc:3037] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.315868 8195 raft_consensus.cc:3037] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.316469 8043 raft_consensus.cc:2454] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f9c688fffd477fae58e802f5367e93 in term 1.
I20260924 00:10:01.316473 8195 raft_consensus.cc:2454] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f9c688fffd477fae58e802f5367e93 in term 1.
I20260924 00:10:01.316632 8266 leader_election.cc:302] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 14f9c688fffd477fae58e802f5367e93; no voters:
I20260924 00:10:01.316761 8415 raft_consensus.cc:2781] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:01.316933 8415 raft_consensus.cc:686] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [term 1 LEADER]: Becoming Leader. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Running, Role: LEADER
I20260924 00:10:01.316991 8415 consensus_queue.cc:237] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.317355 8451 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: TxnStatusTablet state changed. Reason: New leader 14f9c688fffd477fae58e802f5367e93. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.317448 8451 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:10:01.317620 8453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:10:01.317734 8453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:10:01.317744 7898 catalog_manager.cc:5495] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f9c688fffd477fae58e802f5367e93 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } health_report { overall_health: HEALTHY } } }
I20260924 00:10:01.359292 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.356776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.0.98.3:43909)
0924 00:10:01.356827 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.356829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.356839 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:10:01.356924 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.356925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.356945 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.357034 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.357190 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.357679 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.358425 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.358628 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.358705 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.358906 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.359007 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.359092 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.359146 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.359164 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:10:01.359304 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.356700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43909 (local address 127.0.0.1:38068)
0924 00:10:01.356779 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.356793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.356846 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.357038 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.357039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.357040 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:10:01.357111 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.357113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.357767 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.357768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.358339 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.358341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.358705 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.358705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.358779 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.358801 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.359077 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.359086 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.359140 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.359186 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:10:01.360929 8043 raft_consensus.cc:1261] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Refusing update from remote peer 14f9c688fffd477fae58e802f5367e93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.360987 8195 raft_consensus.cc:1261] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Refusing update from remote peer 14f9c688fffd477fae58e802f5367e93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.361215 8415 consensus_queue.cc:1035] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.361332 8451 consensus_queue.cc:1035] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.362010 8427 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: TxnStatusTablet state changed. Reason: New leader 14f9c688fffd477fae58e802f5367e93. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.362112 8427 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.362179 8417 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: TxnStatusTablet state changed. Reason: New leader 14f9c688fffd477fae58e802f5367e93. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.362283 8417 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.363296 8427 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.363401 8427 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 3b3e2ae4412c4273ab84f7542e80939d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.363397 8417 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.363471 8417 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 0a697c8b942c41268fda250d5274f608: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:10:01.364168 8415 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.364281 8415 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:10:01.364166 8451 tablet_replica.cc:439] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14f9c688fffd477fae58e802f5367e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } } }
I20260924 00:10:01.364362 8451 tablet_replica.cc:442] T d67244a67e384510b7082d8e60382be2 P 14f9c688fffd477fae58e802f5367e93: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:10:01.367748 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.365054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.98.1:46189)
0924 00:10:01.365092 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.365094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.365102 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:01.365139 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.365140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.365158 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.365236 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.365328 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.366073 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.366821 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.367127 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.367199 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.367411 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.367540 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.367627 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.367652 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.367672 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:10:01.373231 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.370323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57819 (local address 127.0.98.3:43909)
0924 00:10:01.370374 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.370376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.370539 (+ 163us) server_negotiation.cc:407] Connection header received
0924 00:10:01.370586 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.370587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.370596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.370646 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.370768 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.371440 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.372542 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.372766 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.372819 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.373017 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.373042 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.373080 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.373163 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.373178 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:10:01.373477 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.370226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43909 (local address 127.0.98.1:57819)
0924 00:10:01.370443 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.370455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.370530 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.370650 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.370651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.370652 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:10:01.370711 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.370713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.371530 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.371531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.372415 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.372418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.372861 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.372862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.372924 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.372939 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.373108 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.373109 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.373161 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.373201 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260924 00:10:01.375515 8335 raft_consensus.cc:1261] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Refusing update from remote peer 0a697c8b942c41268fda250d5274f608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.375566 8195 raft_consensus.cc:1261] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Refusing update from remote peer 0a697c8b942c41268fda250d5274f608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.375761 8417 consensus_queue.cc:1035] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.375905 8439 consensus_queue.cc:1035] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.384781 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.381976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46189 (local address 127.0.98.3:60569)
0924 00:10:01.382054 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.382064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:01.382111 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:01.382229 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:01.382230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:01.382231 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:10:01.382291 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.382293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.383133 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.383133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.383969 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:01.383971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.384432 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.384432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:01.384469 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.384482 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.384659 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.384660 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:10:01.384694 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 00:10:01.384725 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 00:10:01.384812 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.382012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60569 (local address 127.0.98.1:46189)
0924 00:10:01.382052 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.382054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.382102 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:10:01.382152 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.382152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.382162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.382245 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.382342 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.383054 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.384059 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.384347 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.384420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.384556 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.384597 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.384656 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.384700 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.384720 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:10:01.393843 7897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.393978 7897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.394640 7897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Sending DeleteTablet for 3 replicas of tablet f4571829b0b945f99721a73154c88043
I20260924 00:10:01.395035 8175 tablet_service.cc:1514] Processing DeleteTablet for tablet f4571829b0b945f99721a73154c88043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260924 00:10:01.395097 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet f4571829b0b945f99721a73154c88043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260924 00:10:01.395135 8433 tablet_replica.cc:330] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: stopping tablet replica
I20260924 00:10:01.395172 8434 tablet_replica.cc:330] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: stopping tablet replica
I20260924 00:10:01.395183 8433 raft_consensus.cc:2229] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.395238 8434 raft_consensus.cc:2229] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:10:01.395251 8433 raft_consensus.cc:2258] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.395326 8434 raft_consensus.cc:2258] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.395411 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet f4571829b0b945f99721a73154c88043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38062
I20260924 00:10:01.395530 8432 tablet_replica.cc:330] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: stopping tablet replica
I20260924 00:10:01.395592 8432 raft_consensus.cc:2229] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.395632 8432 raft_consensus.cc:2258] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.395792 8433 ts_tablet_manager.cc:1905] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.397126 8432 ts_tablet_manager.cc:1905] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.397308 8434 ts_tablet_manager.cc:1905] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.398736 8432 ts_tablet_manager.cc:1918] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:10:01.398775 8434 ts_tablet_manager.cc:1918] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:10:01.398790 8432 log.cc:1196] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/f4571829b0b945f99721a73154c88043
I20260924 00:10:01.398821 8434 log.cc:1196] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/f4571829b0b945f99721a73154c88043
I20260924 00:10:01.399019 8434 ts_tablet_manager.cc:1939] T f4571829b0b945f99721a73154c88043 P 0a697c8b942c41268fda250d5274f608: Deleting consensus metadata
I20260924 00:10:01.399019 8432 ts_tablet_manager.cc:1939] T f4571829b0b945f99721a73154c88043 P 14f9c688fffd477fae58e802f5367e93: Deleting consensus metadata
I20260924 00:10:01.399520 7882 catalog_manager.cc:4841] TS 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189): tablet f4571829b0b945f99721a73154c88043 (table test-table [id=ba0d49e9dc9b46d6af74622c9388594f]) successfully deleted
I20260924 00:10:01.399619 7885 catalog_manager.cc:4841] TS 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909): tablet f4571829b0b945f99721a73154c88043 (table test-table [id=ba0d49e9dc9b46d6af74622c9388594f]) successfully deleted
I20260924 00:10:01.401258 8433 ts_tablet_manager.cc:1918] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:10:01.401327 8433 log.cc:1196] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/f4571829b0b945f99721a73154c88043
I20260924 00:10:01.401533 8433 ts_tablet_manager.cc:1939] T f4571829b0b945f99721a73154c88043 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting consensus metadata
I20260924 00:10:01.401834 7882 catalog_manager.cc:4841] TS 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509): tablet f4571829b0b945f99721a73154c88043 (table test-table [id=ba0d49e9dc9b46d6af74622c9388594f]) successfully deleted
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:10:01.422322 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:01.416662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.0.98.62:34097)
0924 00:10:01.416714 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.416717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.416857 (+ 140us) server_negotiation.cc:407] Connection header received
0924 00:10:01.416933 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.416933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.416944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.417008 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:01.417577 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.418075 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.418835 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.419042 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.419125 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.422253 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57746: BlockingRecv error: recv got EOF from 127.0.0.1:57746 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:10:01.430811 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.423715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.98.62:34097)
0924 00:10:01.423751 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.423752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.423761 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:01.423807 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.423808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.423817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.423865 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:01.424330 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.424818 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.425527 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.425724 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.425798 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.428118 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:01.428119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:01.428120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:01.428128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:01.429663 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.430030 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.430031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.430032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.430092 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:01.430291 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:01.430292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:01.430292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:01.430413 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:01.430526 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.430616 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.430698 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:10:01.432495 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:10:01.432675 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:01.435092 8035 tablet_service.cc:1467] Processing CreateTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e (DEFAULT_TABLE table=test-table [id=767151380b364f40a1a68b60a1ce2654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.435119 8175 tablet_service.cc:1467] Processing CreateTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e (DEFAULT_TABLE table=test-table [id=767151380b364f40a1a68b60a1ce2654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.435297 8175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d877609fea45769f7aeb2e6b7ddc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.435352 8315 tablet_service.cc:1467] Processing CreateTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e (DEFAULT_TABLE table=test-table [id=767151380b364f40a1a68b60a1ce2654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.435461 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d877609fea45769f7aeb2e6b7ddc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.436703 8410 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Bootstrap starting.
I20260924 00:10:01.436703 8409 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Bootstrap starting.
I20260924 00:10:01.437084 8409 tablet_bootstrap.cc:654] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.437100 8410 tablet_bootstrap.cc:654] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.437551 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d877609fea45769f7aeb2e6b7ddc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.438635 8411 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Bootstrap starting.
I20260924 00:10:01.439070 8411 tablet_bootstrap.cc:654] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.439155 8410 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: No bootstrap required, opened a new log
I20260924 00:10:01.439203 8410 ts_tablet_manager.cc:1397] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 00:10:01.439396 8410 raft_consensus.cc:348] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.439471 8410 raft_consensus.cc:374] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.439507 8410 raft_consensus.cc:729] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.439553 8410 consensus_queue.cc:260] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.439669 8410 ts_tablet_manager.cc:1428] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.440109 8409 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: No bootstrap required, opened a new log
I20260924 00:10:01.440199 8409 ts_tablet_manager.cc:1397] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 00:10:01.440306 8409 raft_consensus.cc:348] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.440392 8409 raft_consensus.cc:374] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.440419 8409 raft_consensus.cc:729] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.440460 8409 consensus_queue.cc:260] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.440557 8409 ts_tablet_manager.cc:1428] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.443748 8411 tablet_bootstrap.cc:492] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: No bootstrap required, opened a new log
I20260924 00:10:01.443826 8411 ts_tablet_manager.cc:1397] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 00:10:01.444005 8411 raft_consensus.cc:348] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.444092 8411 raft_consensus.cc:374] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.444113 8411 raft_consensus.cc:729] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.444166 8411 consensus_queue.cc:260] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.444260 8411 ts_tablet_manager.cc:1428] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:01.474255 8427 raft_consensus.cc:482] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:01.474363 8427 raft_consensus.cc:504] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.474524 8427 leader_election.cc:288] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.474841 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d877609fea45769f7aeb2e6b7ddc9e" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93" is_pre_election: true
I20260924 00:10:01.474963 8335 raft_consensus.cc:2454] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 0.
I20260924 00:10:01.475114 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d877609fea45769f7aeb2e6b7ddc9e" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608" is_pre_election: true
I20260924 00:10:01.475193 8043 raft_consensus.cc:2454] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 0.
I20260924 00:10:01.475463 8129 leader_election.cc:302] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f9c688fffd477fae58e802f5367e93, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.475589 8427 raft_consensus.cc:2781] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:01.475620 8427 raft_consensus.cc:482] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:01.475637 8427 raft_consensus.cc:3037] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.476532 8427 raft_consensus.cc:504] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.476658 8427 leader_election.cc:288] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 election: Requested vote from peers 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.476827 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d877609fea45769f7aeb2e6b7ddc9e" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a697c8b942c41268fda250d5274f608"
I20260924 00:10:01.476898 8043 raft_consensus.cc:3037] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.477689 8043 raft_consensus.cc:2454] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 1.
I20260924 00:10:01.477921 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d877609fea45769f7aeb2e6b7ddc9e" candidate_uuid: "3b3e2ae4412c4273ab84f7542e80939d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93"
I20260924 00:10:01.478008 8335 raft_consensus.cc:3037] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.478026 8126 leader_election.cc:302] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.478179 8427 raft_consensus.cc:2781] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:01.478235 8427 raft_consensus.cc:686] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 1 LEADER]: Becoming Leader. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Running, Role: LEADER
I20260924 00:10:01.478276 8427 consensus_queue.cc:237] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.478803 8335 raft_consensus.cc:2454] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3e2ae4412c4273ab84f7542e80939d in term 1.
I20260924 00:10:01.478924 7897 catalog_manager.cc:5495] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3b3e2ae4412c4273ab84f7542e80939d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } health_report { overall_health: UNKNOWN } } }
I20260924 00:10:01.491693 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.489204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.98.2:39509)
0924 00:10:01.489250 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.489253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.489331 (+ 78us) server_negotiation.cc:407] Connection header received
0924 00:10:01.489432 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.489433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.489443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.489499 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.489614 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.490109 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.490911 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.491126 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.491239 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.491419 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.491453 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.491501 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.491573 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.491595 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:10:01.492501 8043 raft_consensus.cc:1261] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Refusing update from remote peer 3b3e2ae4412c4273ab84f7542e80939d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.492527 8335 raft_consensus.cc:1261] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Refusing update from remote peer 3b3e2ae4412c4273ab84f7542e80939d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.492731 8427 consensus_queue.cc:1035] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.492856 8419 consensus_queue.cc:1035] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.495807 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.495950 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.496753 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Sending DeleteTablet for 3 replicas of tablet a4d877609fea45769f7aeb2e6b7ddc9e
I20260924 00:10:01.497128 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38062
I20260924 00:10:01.497128 8175 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260924 00:10:01.497234 8432 tablet_replica.cc:330] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: stopping tablet replica
I20260924 00:10:01.497294 8432 raft_consensus.cc:2229] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.497347 8432 raft_consensus.cc:2258] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.497421 8433 tablet_replica.cc:330] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: stopping tablet replica
I20260924 00:10:01.497475 8433 raft_consensus.cc:2229] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:10:01.497529 8433 raft_consensus.cc:2258] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.497606 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d877609fea45769f7aeb2e6b7ddc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260924 00:10:01.497713 8434 tablet_replica.cc:330] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: stopping tablet replica
I20260924 00:10:01.497772 8434 raft_consensus.cc:2229] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.497839 8434 raft_consensus.cc:2258] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.498134 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:10:01.498296 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:01.498298 8433 ts_tablet_manager.cc:1905] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.498298 8434 ts_tablet_manager.cc:1905] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.499572 8432 ts_tablet_manager.cc:1905] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.499934 8434 ts_tablet_manager.cc:1918] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.499991 8434 log.cc:1196] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/a4d877609fea45769f7aeb2e6b7ddc9e
I20260924 00:10:01.499979 8433 ts_tablet_manager.cc:1918] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.500027 8433 log.cc:1196] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/a4d877609fea45769f7aeb2e6b7ddc9e
I20260924 00:10:01.500217 8434 ts_tablet_manager.cc:1939] T a4d877609fea45769f7aeb2e6b7ddc9e P 0a697c8b942c41268fda250d5274f608: Deleting consensus metadata
I20260924 00:10:01.500226 8433 ts_tablet_manager.cc:1939] T a4d877609fea45769f7aeb2e6b7ddc9e P 3b3e2ae4412c4273ab84f7542e80939d: Deleting consensus metadata
I20260924 00:10:01.500609 7882 catalog_manager.cc:4841] TS 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509): tablet a4d877609fea45769f7aeb2e6b7ddc9e (table test-table [id=767151380b364f40a1a68b60a1ce2654]) successfully deleted
I20260924 00:10:01.501111 8432 ts_tablet_manager.cc:1918] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:01.501164 8432 log.cc:1196] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/a4d877609fea45769f7aeb2e6b7ddc9e
I20260924 00:10:01.501173 8175 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 (DEFAULT_TABLE table=test-table [id=64489806796f46fc98c50ffc4cb6be77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.501204 8315 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 (DEFAULT_TABLE table=test-table [id=64489806796f46fc98c50ffc4cb6be77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.501277 8175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b32272f1f4c36a83688994c1f0931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.501288 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b32272f1f4c36a83688994c1f0931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.501381 8432 ts_tablet_manager.cc:1939] T a4d877609fea45769f7aeb2e6b7ddc9e P 14f9c688fffd477fae58e802f5367e93: Deleting consensus metadata
I20260924 00:10:01.502465 8410 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Bootstrap starting.
I20260924 00:10:01.502770 8410 tablet_bootstrap.cc:654] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.503021 8409 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Bootstrap starting.
I20260924 00:10:01.503361 8409 tablet_bootstrap.cc:654] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.503413 7885 catalog_manager.cc:4841] TS 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909): tablet a4d877609fea45769f7aeb2e6b7ddc9e (table test-table [id=767151380b364f40a1a68b60a1ce2654]) successfully deleted
I20260924 00:10:01.503538 7882 catalog_manager.cc:4841] TS 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189): tablet a4d877609fea45769f7aeb2e6b7ddc9e (table test-table [id=767151380b364f40a1a68b60a1ce2654]) successfully deleted
I20260924 00:10:01.503710 8035 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 (DEFAULT_TABLE table=test-table [id=64489806796f46fc98c50ffc4cb6be77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:01.503815 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b32272f1f4c36a83688994c1f0931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:01.504624 8411 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: Bootstrap starting.
I20260924 00:10:01.504781 8410 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: No bootstrap required, opened a new log
I20260924 00:10:01.504838 8410 ts_tablet_manager.cc:1397] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:10:01.504959 8410 raft_consensus.cc:348] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.505012 8411 tablet_bootstrap.cc:654] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:01.505048 8410 raft_consensus.cc:374] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.505085 8410 raft_consensus.cc:729] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f9c688fffd477fae58e802f5367e93, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.505144 8410 consensus_queue.cc:260] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.505249 8410 ts_tablet_manager.cc:1428] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:01.505604 8409 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: No bootstrap required, opened a new log
I20260924 00:10:01.505661 8409 ts_tablet_manager.cc:1397] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 00:10:01.505694 8411 tablet_bootstrap.cc:492] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: No bootstrap required, opened a new log
I20260924 00:10:01.505746 8411 ts_tablet_manager.cc:1397] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:10:01.505788 8409 raft_consensus.cc:348] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.505879 8409 raft_consensus.cc:374] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.505903 8409 raft_consensus.cc:729] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3e2ae4412c4273ab84f7542e80939d, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.505896 8411 raft_consensus.cc:348] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.505970 8411 raft_consensus.cc:374] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:01.505944 8409 consensus_queue.cc:260] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.505997 8411 raft_consensus.cc:729] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Initialized, Role: FOLLOWER
I20260924 00:10:01.506039 8409 ts_tablet_manager.cc:1428] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:01.506035 8411 consensus_queue.cc:260] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.506121 8411 ts_tablet_manager.cc:1428] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:01.635773 8439 raft_consensus.cc:482] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:01.635915 8439 raft_consensus.cc:504] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.636137 8439 leader_election.cc:288] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.636397 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b32272f1f4c36a83688994c1f0931" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93" is_pre_election: true
I20260924 00:10:01.636397 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b32272f1f4c36a83688994c1f0931" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d" is_pre_election: true
I20260924 00:10:01.636502 8195 raft_consensus.cc:2454] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 0.
I20260924 00:10:01.636502 8335 raft_consensus.cc:2454] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 0.
I20260924 00:10:01.636677 7986 leader_election.cc:302] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.636801 8439 raft_consensus.cc:2781] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:01.636845 8439 raft_consensus.cc:482] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:01.636868 8439 raft_consensus.cc:3037] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.637562 8439 raft_consensus.cc:504] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.637713 8439 leader_election.cc:288] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 election: Requested vote from peers 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509), 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909)
I20260924 00:10:01.637869 8195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b32272f1f4c36a83688994c1f0931" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3e2ae4412c4273ab84f7542e80939d"
I20260924 00:10:01.637892 8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b32272f1f4c36a83688994c1f0931" candidate_uuid: "0a697c8b942c41268fda250d5274f608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f9c688fffd477fae58e802f5367e93"
I20260924 00:10:01.637961 8335 raft_consensus.cc:3037] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.637961 8195 raft_consensus.cc:3037] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:01.638710 8335 raft_consensus.cc:2454] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 1.
I20260924 00:10:01.638710 8195 raft_consensus.cc:2454] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a697c8b942c41268fda250d5274f608 in term 1.
I20260924 00:10:01.638890 7986 leader_election.cc:302] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a697c8b942c41268fda250d5274f608, 3b3e2ae4412c4273ab84f7542e80939d; no voters:
I20260924 00:10:01.638983 8439 raft_consensus.cc:2781] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:01.639066 8439 raft_consensus.cc:686] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 1 LEADER]: Becoming Leader. State: Replica: 0a697c8b942c41268fda250d5274f608, State: Running, Role: LEADER
I20260924 00:10:01.639122 8439 consensus_queue.cc:237] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } }
I20260924 00:10:01.639786 7896 catalog_manager.cc:5495] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a697c8b942c41268fda250d5274f608 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0a697c8b942c41268fda250d5274f608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a697c8b942c41268fda250d5274f608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 } health_report { overall_health: UNKNOWN } } }
I20260924 00:10:01.658869 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.656912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.0.98.1:46189)
0924 00:10:01.656955 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:01.656957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:01.656966 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:01.657026 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:01.657027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:01.657035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:01.657086 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:01.657169 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.657638 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.658348 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:01.658519 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:01.658569 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:01.658686 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:01.658712 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:01.658750 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:01.658798 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 00:10:01.658813 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:10:01.661113 8335 raft_consensus.cc:1261] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Refusing update from remote peer 0a697c8b942c41268fda250d5274f608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.661134 8195 raft_consensus.cc:1261] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Refusing update from remote peer 0a697c8b942c41268fda250d5274f608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:01.661334 8464 consensus_queue.cc:1035] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f9c688fffd477fae58e802f5367e93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.661504 8439 consensus_queue.cc:1035] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3e2ae4412c4273ab84f7542e80939d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:01.672646 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.672771 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:01.673391 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5b6a5840ce2400d930eaecb91073433: Sending DeleteTablet for 3 replicas of tablet 5f3b32272f1f4c36a83688994c1f0931
I20260924 00:10:01.673676 8035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260924 00:10:01.673674 8175 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260924 00:10:01.673786 8433 tablet_replica.cc:330] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: stopping tablet replica
I20260924 00:10:01.673789 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b32272f1f4c36a83688994c1f0931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38062
I20260924 00:10:01.673851 8433 raft_consensus.cc:2229] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.673877 8434 tablet_replica.cc:330] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: stopping tablet replica
I20260924 00:10:01.673877 8432 tablet_replica.cc:330] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: stopping tablet replica
I20260924 00:10:01.673888 8433 raft_consensus.cc:2258] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.673935 8434 raft_consensus.cc:2229] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:10:01.673935 8432 raft_consensus.cc:2229] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:01.673977 8432 raft_consensus.cc:2258] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.673995 8434 raft_consensus.cc:2258] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:01.674197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7970
I20260924 00:10:01.674247 8433 ts_tablet_manager.cc:1905] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.674283 8432 ts_tablet_manager.cc:1905] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.674350 8434 ts_tablet_manager.cc:1905] T 5f3b32272f1f4c36a83688994c1f0931 P 0a697c8b942c41268fda250d5274f608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:01.675925 8433 ts_tablet_manager.cc:1918] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:10:01.675983 8433 log.cc:1196] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/5f3b32272f1f4c36a83688994c1f0931
I20260924 00:10:01.676200 8433 ts_tablet_manager.cc:1939] T 5f3b32272f1f4c36a83688994c1f0931 P 3b3e2ae4412c4273ab84f7542e80939d: Deleting consensus metadata
I20260924 00:10:01.676379 8432 ts_tablet_manager.cc:1918] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:10:01.676441 8432 log.cc:1196] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/5f3b32272f1f4c36a83688994c1f0931
I20260924 00:10:01.676573 7882 catalog_manager.cc:4841] TS 3b3e2ae4412c4273ab84f7542e80939d (127.0.98.2:39509): tablet 5f3b32272f1f4c36a83688994c1f0931 (table test-table [id=64489806796f46fc98c50ffc4cb6be77]) successfully deleted
I20260924 00:10:01.676651 8432 ts_tablet_manager.cc:1939] T 5f3b32272f1f4c36a83688994c1f0931 P 14f9c688fffd477fae58e802f5367e93: Deleting consensus metadata
I20260924 00:10:01.676954 7885 catalog_manager.cc:4841] TS 14f9c688fffd477fae58e802f5367e93 (127.0.98.3:43909): tablet 5f3b32272f1f4c36a83688994c1f0931 (table test-table [id=64489806796f46fc98c50ffc4cb6be77]) successfully deleted
W20260924 00:10:01.680985 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46189 (error 108)
I20260924 00:10:01.681169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8110
I20260924 00:10:01.681559 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:01.681373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46189 (local address 127.0.0.1:46946)
0924 00:10:01.681476 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:01.681514 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":38,"threads_started":1}
W20260924 00:10:01.681638 7882 catalog_manager.cc:4584] TS 0a697c8b942c41268fda250d5274f608 (127.0.98.1:46189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f3b32272f1f4c36a83688994c1f0931: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46189: connect: Connection refused (error 111)
I20260924 00:10:01.687422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8250
I20260924 00:10:01.693353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 7866
2026-09-24T00:10:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2542 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[8498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[8498](info): set up 2 sockets
Sep 24 00:10:01 dist-test-slave-3q8k krb5kdc[8498](info): commencing operation
krb5kdc: starting...
W20260924 00:10:02.740047 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:10:02 dist-test-slave-3q8k krb5kdc[8498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:10:02.867372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:02.950771 8511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:02.951045 8511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:02.951086 8511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:02.952294 8511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:02.952373 8511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:02.952396 8511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:02.952417 8511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:02.952445 8511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:02.954064 8511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:02.954375 8511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:02.954607 8511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:02.959560 8517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:02.959623 8519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:02.959563 8516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:02.959914 8511 server_base.cc:1034] running on GCE node
I20260924 00:10:02.960162 8511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:02.960480 8511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:02.961663 8511 hybrid_clock.cc:648] HybridClock initialized: now 1790208602961657 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T00:10:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1270 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:10:03 dist-test-slave-3q8k krb5kdc[8524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:10:03 dist-test-slave-3q8k krb5kdc[8524](info): set up 2 sockets
Sep 24 00:10:03 dist-test-slave-3q8k krb5kdc[8524](info): commencing operation
krb5kdc: starting...
W20260924 00:10:04.063840 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:10:04.182322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:04.263154 8537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:04.263425 8537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:04.263463 8537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:04.264642 8537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:04.264714 8537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:04.264741 8537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:04.264770 8537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:04.264796 8537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:04.266306 8537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:04.266583 8537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:04.266817 8537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:04.271837 8543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.271813 8545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.271821 8542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:04.271852 8537 server_base.cc:1034] running on GCE node
I20260924 00:10:04.272259 8537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:04.272503 8537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:04.273679 8537 hybrid_clock.cc:648] HybridClock initialized: now 1790208604273663 us; error 30 us; skew 500 ppm
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:04.276494 8537 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:10:04.276969 8537 webserver.cc:460] Webserver started at http://127.0.98.62:37623/ using document root <none> and password file <none>
I20260924 00:10:04.277211 8537 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:04.277257 8537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:04.277433 8537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:04.278505 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "415fad1531e04a11a5cbd53e69632725"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "f23d1f96f2e54a56c711633945602c85"
server_key_iv: "c6d70993de5e26b410139f955bbf8aeb"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.278887 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "415fad1531e04a11a5cbd53e69632725"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "f23d1f96f2e54a56c711633945602c85"
server_key_iv: "c6d70993de5e26b410139f955bbf8aeb"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.280344 8537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:04.281122 8552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.281303 8537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:04.281369 8537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "415fad1531e04a11a5cbd53e69632725"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "f23d1f96f2e54a56c711633945602c85"
server_key_iv: "c6d70993de5e26b410139f955bbf8aeb"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.281464 8537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:04.323313 8537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:04.325835 8537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:04.325955 8537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:04.329722 8537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34829
I20260924 00:10:04.329810 8604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34829 every 8 connection(s)
I20260924 00:10:04.330312 8537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:04.331056 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:04.333474 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Bootstrap starting.
I20260924 00:10:04.334100 8605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:04.334407 8605 log.cc:824] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:04.335140 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: No bootstrap required, opened a new log
I20260924 00:10:04.336792 8605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:04.336936 8605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:04.336958 8605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415fad1531e04a11a5cbd53e69632725, State: Initialized, Role: FOLLOWER
I20260924 00:10:04.337116 8605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:04.337200 8605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:04.337247 8605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:04.337319 8605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:04.337481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8537
I20260924 00:10:04.337590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:10:04.337751 392 external_mini_cluster.cc:1428] Setting key d81735bcd8cf607ced3b49136f4a06af
I20260924 00:10:04.338492 8605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:04.338647 8605 leader_election.cc:302] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415fad1531e04a11a5cbd53e69632725; no voters:
I20260924 00:10:04.338833 8605 leader_election.cc:288] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:10:04.338883 8610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:04.339084 8610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 LEADER]: Becoming Leader. State: Replica: 415fad1531e04a11a5cbd53e69632725, State: Running, Role: LEADER
I20260924 00:10:04.339206 8605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:04.339241 8610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:04.339641 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415fad1531e04a11a5cbd53e69632725. Latest consensus state: current_term: 1 leader_uuid: "415fad1531e04a11a5cbd53e69632725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260924 00:10:04.339735 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:04.339931 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:04.340469 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:04.342401 8611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415fad1531e04a11a5cbd53e69632725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260924 00:10:04.342485 8611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:04.342983 8614 catalog_manager.cc:1347] Generated new cluster ID: 31d3a95c2c1c45deb3ce91175c355085
I20260924 00:10:04.343037 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:04.348489 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.338310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.62:34829)
0924 00:10:04.338496 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:04.338504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:04.338536 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:10:04.338654 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:04.338662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:04.338722 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:04.338877 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:04.339602 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.341012 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.342199 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.342707 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.342823 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.345216 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:04.345221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:04.345221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:04.345232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:04.346905 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.347321 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.347325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.347325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.347433 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.347655 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.347656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.347656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.347894 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:04.348088 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:04.348166 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:10:04.348313 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260924 00:10:04.355535 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:10:04.356251 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:04.362048 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Generated new TSK 0
I20260924 00:10:04.362259 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:10:04.390978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:04.479959 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:04.480228 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:04.480274 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:04.481501 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:04.481580 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:04.481630 8631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:04.483273 8631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:04.483560 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:04.483853 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:04.489315 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.489317 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.489370 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:04.490381 8631 server_base.cc:1034] running on GCE node
I20260924 00:10:04.490670 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:04.490954 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:04.492144 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790208604492131 us; error 33 us; skew 500 ppm
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:04.494779 8631 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:10:04.495466 8631 webserver.cc:460] Webserver started at http://127.0.98.1:37817/ using document root <none> and password file <none>
I20260924 00:10:04.495729 8631 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:04.495803 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:04.495957 8631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:04.497555 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "0089daa271c346da98458ab27041316d"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "b26d8ad4d1e6038796355977c5bce5e1"
server_key_iv: "817853913335455f75f0de21ab79016b"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.498054 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0089daa271c346da98458ab27041316d"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "b26d8ad4d1e6038796355977c5bce5e1"
server_key_iv: "817853913335455f75f0de21ab79016b"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.500191 8631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 00:10:04.501277 8646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.501519 8631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:04.501626 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "0089daa271c346da98458ab27041316d"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "b26d8ad4d1e6038796355977c5bce5e1"
server_key_iv: "817853913335455f75f0de21ab79016b"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.501716 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:04.527820 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:04.530292 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:04.530411 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:04.530606 8631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:04.530946 8631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:04.530984 8631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.531008 8631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:04.531064 8631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.537449 8631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33325
I20260924 00:10:04.537490 8759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33325 every 8 connection(s)
I20260924 00:10:04.537833 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:04.546708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8631
I20260924 00:10:04.546829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:10:04.547021 392 external_mini_cluster.cc:1428] Setting key 9847a0fefbcc29adbc1f735def96cfcb
I20260924 00:10:04.547529 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.538630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55517 (local address 127.0.98.62:34829)
0924 00:10:04.538691 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:04.538693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:04.538898 (+ 205us) server_negotiation.cc:407] Connection header received
0924 00:10:04.540113 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:04.540114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:04.540131 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:04.540181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:04.540867 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.541383 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.542395 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.542603 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.542692 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.544876 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:04.544878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:04.544878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:04.544886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:04.546080 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.546448 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.546449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.546450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.546499 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.546750 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.546751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.546751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.546908 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:04.546970 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:04.547367 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 00:10:04.547464 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:10:04.547745 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.538620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:55517)
0924 00:10:04.538773 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:04.538809 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:04.539957 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:04.540272 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:04.540275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:04.540544 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:04.540779 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.540783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.541482 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.541483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.542291 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.542293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.542699 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.542700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.542825 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.543418 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:04.543426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:04.544778 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:04.546159 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.546164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.546166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.546380 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.546556 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.546557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.546557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.546660 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.546974 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:04.546981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:04.547161 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 00:10:04.547329 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 00:10:04.547518 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 00:10:04.549247 8760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260924 00:10:04.549366 8760 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:04.549576 8760 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260924 00:10:04.550179 8569 ts_manager.cc:194] Registered new tserver with Master: 0089daa271c346da98458ab27041316d (127.0.98.1:33325)
I20260924 00:10:04.550802 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55517
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:10:04.573915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:04.657112 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:04.657363 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:04.657405 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:04.658573 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:04.658646 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:04.658696 8765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:04.660434 8765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:04.660724 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:04.660964 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:04.665827 8771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.665879 8773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.665982 8770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:04.666236 8765 server_base.cc:1034] running on GCE node
I20260924 00:10:04.666467 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:04.666720 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:04.667884 8765 hybrid_clock.cc:648] HybridClock initialized: now 1790208604667871 us; error 46 us; skew 500 ppm
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:04.670477 8765 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:10:04.670935 8765 webserver.cc:460] Webserver started at http://127.0.98.2:39957/ using document root <none> and password file <none>
I20260924 00:10:04.671350 8765 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:04.671422 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:04.671587 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:04.672647 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "cf31ea5f6515416daf0d6fb636ebb6ad"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "7743266d5a102f2d23452f43d731d643"
server_key_iv: "f871ca9e1b3b34ab66923b3033334252"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.673028 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cf31ea5f6515416daf0d6fb636ebb6ad"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "7743266d5a102f2d23452f43d731d643"
server_key_iv: "f871ca9e1b3b34ab66923b3033334252"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.674328 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:04.675012 8780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.675186 8765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:04.675315 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "cf31ea5f6515416daf0d6fb636ebb6ad"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "7743266d5a102f2d23452f43d731d643"
server_key_iv: "f871ca9e1b3b34ab66923b3033334252"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.675405 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:04.692351 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:04.694506 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:04.694711 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:04.694977 8765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:04.695412 8765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:04.695479 8765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.695525 8765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:04.695552 8765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.701354 8765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38539
I20260924 00:10:04.701427 8893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38539 every 8 connection(s)
I20260924 00:10:04.701746 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:04.708812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8765
I20260924 00:10:04.708928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:10:04.709128 392 external_mini_cluster.cc:1428] Setting key 5d690c47703a0507096f0569fd1bfc69
I20260924 00:10:04.711330 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.702476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40043 (local address 127.0.98.62:34829)
0924 00:10:04.702546 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:04.702549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:04.702761 (+ 212us) server_negotiation.cc:407] Connection header received
0924 00:10:04.703781 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:04.703782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:04.703792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:04.703844 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:04.704424 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.704908 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.706000 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.706191 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.706246 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.708379 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:04.708380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:04.708380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:04.708387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:04.709928 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.710321 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.710322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.710323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.710386 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.710620 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.710621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.710621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.710764 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:04.710831 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:04.711078 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 00:10:04.711168 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:10:04.711539 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.702469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:40043)
0924 00:10:04.702634 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:04.702669 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:04.703683 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:04.703897 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:04.703900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:04.704161 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:04.704348 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.704353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.704989 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.704990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.705924 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.705927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.706265 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.706265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.706359 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.706895 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:04.706903 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:04.708298 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:04.710006 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.710011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.710013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.710243 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.710460 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.710461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.710461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.710557 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.710831 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:04.710834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:04.710950 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 00:10:04.711032 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:10:04.711257 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 00:10:04.711992 8894 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260924 00:10:04.712109 8894 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:04.712297 8894 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260924 00:10:04.712709 8569 ts_manager.cc:194] Registered new tserver with Master: cf31ea5f6515416daf0d6fb636ebb6ad (127.0.98.2:38539)
I20260924 00:10:04.713095 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40043
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:10:04.734424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:04.824415 8899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:04.824699 8899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:04.824743 8899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:04.826089 8899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:04.826179 8899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:04.826231 8899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:04.828056 8899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:04.828363 8899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:04.828621 8899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:04.834072 8905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.834121 8907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:04.834105 8904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:04.834367 8899 server_base.cc:1034] running on GCE node
I20260924 00:10:04.834640 8899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:04.834889 8899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:04.836043 8899 hybrid_clock.cc:648] HybridClock initialized: now 1790208604836026 us; error 32 us; skew 500 ppm
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:04.838510 8899 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:10:04.838891 8899 webserver.cc:460] Webserver started at http://127.0.98.3:36425/ using document root <none> and password file <none>
I20260924 00:10:04.839093 8899 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:04.839134 8899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:04.839285 8899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:04.840457 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "1c761d16db3c481c8ce28893001744c9"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "0fb633465e8fab7532a14d5311cf8797"
server_key_iv: "a45316ee091114850cc1825995a633fd"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.840806 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1c761d16db3c481c8ce28893001744c9"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "0fb633465e8fab7532a14d5311cf8797"
server_key_iv: "a45316ee091114850cc1825995a633fd"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.842170 8899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:04.842826 8914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.843010 8899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:04.843086 8899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "1c761d16db3c481c8ce28893001744c9"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "0fb633465e8fab7532a14d5311cf8797"
server_key_iv: "a45316ee091114850cc1825995a633fd"
server_key_version: "encryptionkey@0"
I20260924 00:10:04.843135 8899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:04.867449 8899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:04.869922 8899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:04.870044 8899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:04.870234 8899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:04.870556 8899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:04.870592 8899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.870615 8899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:04.870663 8899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:04.876497 8899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38771
I20260924 00:10:04.876546 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38771 every 8 connection(s)
I20260924 00:10:04.876850 8899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:10:04.879935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 8899
I20260924 00:10:04.880033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:10:04.880201 392 external_mini_cluster.cc:1428] Setting key 259c196c74a5815f188b67793be5adbd
Sep 24 00:10:04 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:04.886786 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.877570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45311 (local address 127.0.98.62:34829)
0924 00:10:04.877640 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:04.877642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:04.877860 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:10:04.878939 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:04.878940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:04.878952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:04.879008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:04.879727 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.880518 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.881687 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.881867 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.881920 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.884085 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:04.884086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:04.884086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:04.884093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:04.885512 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.885960 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.885961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.885961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.886012 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:04.886275 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:04.886275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:04.886275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:04.886373 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:04.886421 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:04.886662 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 00:10:04.886721 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:10:04.887019 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:04.877560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:45311)
0924 00:10:04.877725 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:04.877783 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:04.878833 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:04.879085 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:04.879089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:04.879422 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:04.879640 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.879647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.880627 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.880628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.881596 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:04.881599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:04.881941 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:04.881941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:04.882046 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:04.882488 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:04.882496 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:04.883920 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:04.885583 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.885588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.885590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.885893 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.886077 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:04.886078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:04.886078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:04.886210 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:04.886426 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:04.886430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:04.886545 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 00:10:04.886636 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 00:10:04.886753 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260924 00:10:04.887568 9028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260924 00:10:04.887661 9028 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:04.887859 9028 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260924 00:10:04.888266 8569 ts_manager.cc:194] Registered new tserver with Master: 1c761d16db3c481c8ce28893001744c9 (127.0.98.3:38771)
I20260924 00:10:04.888624 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45311
I20260924 00:10:04.891880 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:10:04.966132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8631
I20260924 00:10:04.970835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8765
I20260924 00:10:04.975029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8899
I20260924 00:10:04.979465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 8537
I20260924 00:10:04.984632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--webserver_interface=127.0.98.62
--webserver_port=37623
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:05.067232 9037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:05.067444 9037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:05.067466 9037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:05.068683 9037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:05.068739 9037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:05.068754 9037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:05.068768 9037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:05.068779 9037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:05.070461 9037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:05.070744 9037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:05.071031 9037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:05.076320 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.076368 9043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.076368 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:05.076601 9037 server_base.cc:1034] running on GCE node
I20260924 00:10:05.076853 9037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:05.077171 9037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:05.078364 9037 hybrid_clock.cc:648] HybridClock initialized: now 1790208605078328 us; error 48 us; skew 500 ppm
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:05.080971 9037 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:10:05.081427 9037 webserver.cc:460] Webserver started at http://127.0.98.62:37623/ using document root <none> and password file <none>
I20260924 00:10:05.081676 9037 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:05.081760 9037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:05.083138 9037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:05.083827 9052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.083989 9037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.084095 9037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "415fad1531e04a11a5cbd53e69632725"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "f23d1f96f2e54a56c711633945602c85"
server_key_iv: "c6d70993de5e26b410139f955bbf8aeb"
server_key_version: "encryptionkey@0"
I20260924 00:10:05.084460 9037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:05.092792 9037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:05.094543 9037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:05.094715 9037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:05.098824 9037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34829
I20260924 00:10:05.098855 9104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34829 every 8 connection(s)
I20260924 00:10:05.099442 9037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:05.100649 9105 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:10:05.101279 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Bootstrap starting.
I20260924 00:10:05.109246 9105 log.cc:824] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:05.109586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9037
I20260924 00:10:05.109875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=37817
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:10:05.111135 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:10:05.111449 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Bootstrap complete.
I20260924 00:10:05.113981 9105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:05.114223 9105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415fad1531e04a11a5cbd53e69632725, State: Initialized, Role: FOLLOWER
I20260924 00:10:05.114392 9105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:05.114472 9105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:05.114533 9105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:05.114583 9105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:10:05.115643 9105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:05.115764 9105 leader_election.cc:302] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415fad1531e04a11a5cbd53e69632725; no voters:
I20260924 00:10:05.115957 9105 leader_election.cc:288] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:10:05.115993 9109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:10:05.116115 9109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [term 2 LEADER]: Becoming Leader. State: Replica: 415fad1531e04a11a5cbd53e69632725, State: Running, Role: LEADER
I20260924 00:10:05.116197 9109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260924 00:10:05.116225 9105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:05.116503 9109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415fad1531e04a11a5cbd53e69632725. Latest consensus state: current_term: 2 leader_uuid: "415fad1531e04a11a5cbd53e69632725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260924 00:10:05.116577 9109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:05.116508 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "415fad1531e04a11a5cbd53e69632725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415fad1531e04a11a5cbd53e69632725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260924 00:10:05.116641 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:05.117992 9125 catalog_manager.cc:1259] Loaded cluster ID: 31d3a95c2c1c45deb3ce91175c355085
I20260924 00:10:05.118052 9125 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: loading cluster ID for follower catalog manager: success
I20260924 00:10:05.119181 9125 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: acquiring CA information for follower catalog manager: success
I20260924 00:10:05.119526 9125 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:10:05.119714 9114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:05.119884 9114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:05.119956 9114 catalog_manager.cc:1259] Loaded cluster ID: 31d3a95c2c1c45deb3ce91175c355085
I20260924 00:10:05.119992 9114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:10:05.120190 9114 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:05.120277 9114 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 415fad1531e04a11a5cbd53e69632725: Loaded TSK: 0
I20260924 00:10:05.120412 9114 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:10:05.193994 9108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:05.194247 9108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:05.194290 9108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:05.195581 9108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:05.195631 9108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:05.195659 9108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:05.197115 9108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37817
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:05.197363 9108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:05.197613 9108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:05.203478 9132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.203488 9131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.203668 9134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:05.203855 9108 server_base.cc:1034] running on GCE node
I20260924 00:10:05.204070 9108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:05.204361 9108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:05.205516 9108 hybrid_clock.cc:648] HybridClock initialized: now 1790208605205505 us; error 27 us; skew 500 ppm
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208605, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:05.208230 9108 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:10:05.208666 9108 webserver.cc:460] Webserver started at http://127.0.98.1:37817/ using document root <none> and password file <none>
I20260924 00:10:05.208902 9108 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:05.208985 9108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:05.210400 9108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:10:05.211058 9141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.211297 9108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:10:05.211400 9108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "0089daa271c346da98458ab27041316d"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "b26d8ad4d1e6038796355977c5bce5e1"
server_key_iv: "817853913335455f75f0de21ab79016b"
server_key_version: "encryptionkey@0"
I20260924 00:10:05.211767 9108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:05.226369 9108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:05.228849 9108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:05.229024 9108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:05.229285 9108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:05.229600 9108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:05.229665 9108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.229707 9108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:05.229732 9108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.236439 9108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33325
I20260924 00:10:05.236474 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33325 every 8 connection(s)
I20260924 00:10:05.236899 9108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:10:05.245159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9108
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.245468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38539
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=39957
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.246855 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.237720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:33191)
0924 00:10:05.237909 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.237977 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.238602 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.239342 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.239345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.239932 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.240128 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.240133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.241881 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.241881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.242701 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.242704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.243523 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.243524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.243635 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.244218 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.244226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.246654 (+ 2428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
W20260924 00:10:05.247092 9255 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 00:10:05.247159 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.237814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33191 (local address 127.0.98.62:34829)
0924 00:10:05.238841 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.238850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.238889 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:10:05.239027 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.239033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.239087 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.239245 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.240281 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.241767 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.242802 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.243429 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.243528 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.246968 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33191: BlockingRecv error: recv got EOF from 127.0.98.1:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":89,"threads_started":1}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.251943 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.247314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:50351)
0924 00:10:05.247442 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.247452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.247502 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.247664 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.247664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.247820 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.247872 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.247873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.248674 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.248675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.249188 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.249189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.249576 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.249577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.249653 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.250101 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.250106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.251850 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260924 00:10:05.251991 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.247419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50351 (local address 127.0.98.62:34829)
0924 00:10:05.247454 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.247456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.247537 (+ 81us) server_negotiation.cc:407] Connection header received
0924 00:10:05.247564 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.247564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.247579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.247651 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.247974 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.248583 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.249291 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.249492 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.249567 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.251926 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50351: BlockingRecv error: recv got EOF from 127.0.98.1:50351 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.256749 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.252475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:42797)
0924 00:10:05.252519 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.252529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.252568 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.252707 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.252708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.252834 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.252861 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.252862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.253468 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.253469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.253966 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.253967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.254320 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.254321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.254372 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.254938 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.254942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.256661 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:10:05.256799 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.252522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42797 (local address 127.0.98.62:34829)
0924 00:10:05.252559 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.252562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.252569 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:10:05.252637 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.252638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.252648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.252698 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.252934 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.253388 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.254052 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.254245 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.254323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.256735 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42797: BlockingRecv error: recv got EOF from 127.0.98.1:42797 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 00:10:05.256887 9255 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:10:05.329068 9259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:05.329277 9259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:05.329300 9259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:05.330511 9259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:05.330562 9259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:05.330590 9259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:05.332147 9259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39957
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:05.332408 9259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:05.332655 9259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:05.337821 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.337958 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.337849 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:05.338008 9259 server_base.cc:1034] running on GCE node
I20260924 00:10:05.338205 9259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:05.338464 9259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:05.339633 9259 hybrid_clock.cc:648] HybridClock initialized: now 1790208605339632 us; error 50 us; skew 500 ppm
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208605, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:05.342025 9259 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:10:05.342468 9259 webserver.cc:460] Webserver started at http://127.0.98.2:39957/ using document root <none> and password file <none>
I20260924 00:10:05.342715 9259 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:05.342800 9259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:05.344213 9259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:05.344776 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.344933 9259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:05.345033 9259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "cf31ea5f6515416daf0d6fb636ebb6ad"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "7743266d5a102f2d23452f43d731d643"
server_key_iv: "f871ca9e1b3b34ab66923b3033334252"
server_key_version: "encryptionkey@0"
I20260924 00:10:05.345391 9259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:05.356346 9259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:05.358281 9259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:05.358453 9259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:05.358692 9259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:05.358996 9259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:05.359052 9259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.359093 9259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:05.359117 9259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.365481 9259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38539
I20260924 00:10:05.365526 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38539 every 8 connection(s)
I20260924 00:10:05.365994 9259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:10:05.371544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9259
I20260924 00:10:05.371851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38771
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=36425
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.373900 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.366860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:60361)
0924 00:10:05.367026 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.367072 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.367646 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.367880 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.367883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.368460 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.368659 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.368664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.369300 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.369301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.370096 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.370098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.370429 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.370430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.370528 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.371110 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.371118 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.373754 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260924 00:10:05.373991 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.366881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60361 (local address 127.0.98.62:34829)
0924 00:10:05.366961 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.366964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.367162 (+ 198us) server_negotiation.cc:407] Connection header received
0924 00:10:05.367750 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.367752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.367764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.367827 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.368741 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.369219 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.370173 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.370356 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.370407 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.373896 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60361: BlockingRecv error: recv got EOF from 127.0.98.2:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 00:10:05.374171 9389 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.379534 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.374380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:33681)
0924 00:10:05.374431 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.374442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.374481 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.374602 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.374602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.374730 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.374772 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.374774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.375567 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.375568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.376544 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.376546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.376898 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.376899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.376984 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.377411 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.377417 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.379474 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:10:05.379679 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.374446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33681 (local address 127.0.98.62:34829)
0924 00:10:05.374504 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.374507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.374526 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:10:05.374547 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.374548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.374557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.374601 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.374851 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.375423 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.376640 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.376816 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.376872 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.379591 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33681: BlockingRecv error: recv got EOF from 127.0.98.2:33681 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.384409 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.379857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:39037)
0924 00:10:05.379908 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.379917 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.379950 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.380100 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.380101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.380207 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.380237 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.380238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.381196 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.381197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.381681 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.381683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.382181 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.382181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.382216 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.382692 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.382696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.384358 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 00:10:05.384636 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.379929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39037 (local address 127.0.98.62:34829)
0924 00:10:05.379978 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.379980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.379988 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:05.380012 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.380013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.380024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.380090 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.380311 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.381096 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.381768 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.382102 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.382181 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.384475 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39037: BlockingRecv error: recv got EOF from 127.0.98.2:39037 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 00:10:05.384666 9389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:10:05.455506 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:05.455705 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:05.455727 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:05.456888 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:05.456933 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:05.456988 9392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:05.458402 9392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36425
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:05.458652 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:05.458885 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:05.464044 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.464102 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:05.464056 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:05.464388 9392 server_base.cc:1034] running on GCE node
I20260924 00:10:05.464592 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:05.464790 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:05.465930 9392 hybrid_clock.cc:648] HybridClock initialized: now 1790208605465919 us; error 29 us; skew 500 ppm
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208605, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:05.468571 9392 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:10:05.469054 9392 webserver.cc:460] Webserver started at http://127.0.98.3:36425/ using document root <none> and password file <none>
I20260924 00:10:05.469326 9392 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:05.469413 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:05.470821 9392 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:10:05.471527 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.471707 9392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:05.471807 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "1c761d16db3c481c8ce28893001744c9"
format_stamp: "Formatted at 2026-09-24 00:10:04 on dist-test-slave-3q8k"
server_key: "0fb633465e8fab7532a14d5311cf8797"
server_key_iv: "a45316ee091114850cc1825995a633fd"
server_key_version: "encryptionkey@0"
I20260924 00:10:05.472170 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:05.491995 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:05.494029 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:05.494215 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:05.494467 9392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:05.494786 9392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:05.494845 9392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.494889 9392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:05.494911 9392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:05.501402 9392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38771
I20260924 00:10:05.501441 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38771 every 8 connection(s)
I20260924 00:10:05.501811 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:10:05.507449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9392
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:10:05.509430 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.502567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37499 (local address 127.0.98.62:34829)
0924 00:10:05.502615 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.502617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.502826 (+ 209us) server_negotiation.cc:407] Connection header received
0924 00:10:05.503401 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.503402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.503413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.503465 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.504381 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.504886 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.505895 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.506087 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.506150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.509372 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37499: BlockingRecv error: recv got EOF from 127.0.98.3:37499 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:10:05.509517 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.502576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:37499)
0924 00:10:05.502710 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.502747 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.503303 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.503523 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.503527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.504087 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.504296 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.504300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.504981 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.504982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.505809 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.505811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.506178 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.506178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.506279 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.506766 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.506774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.509303 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
W20260924 00:10:05.509791 9521 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.514707 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.510030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:45843)
0924 00:10:05.510084 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.510094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.510135 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.510261 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.510262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.510388 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.510452 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.510454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.511185 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.511185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.511785 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.511787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.512273 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.512274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.512342 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.512864 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.512869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.514621 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 00:10:05.514751 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.510082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45843 (local address 127.0.98.62:34829)
0924 00:10:05.510124 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.510126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.510135 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:05.510194 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.510194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.510204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.510257 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.510519 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.511102 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.511880 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.512194 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.512272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.514693 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45843: BlockingRecv error: recv got EOF from 127.0.98.3:45843 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:10:05.515648 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.508022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.62:34829)
0924 00:10:05.508157 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.508159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.508168 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:05.508191 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.508192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.508203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.508276 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.508926 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.509414 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.510470 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.510713 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.510763 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.512155 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:05.512159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:05.512160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:05.512170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:05.513976 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:05.514359 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:05.514363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:05.514363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:05.514463 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:05.514689 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:05.514690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:05.514690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:05.514982 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:05.515139 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:05.515414 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 00:10:05.515539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:05 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:05.519493 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:05.515124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:55013)
0924 00:10:05.515236 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:05.515247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:05.515301 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:05.515441 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:05.515442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:05.515586 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:05.515622 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.515623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.516503 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.516505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.517008 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:05.517010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.517392 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.517392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:05.517429 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.517795 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:05.517798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:05.519406 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 00:10:05.519550 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:05.515178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55013 (local address 127.0.98.62:34829)
0924 00:10:05.515240 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:05.515241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:05.515283 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:10:05.515369 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:05.515370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:05.515391 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:05.515440 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:05.515708 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.516393 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.517078 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:05.517327 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:05.517381 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:05.519480 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55013: BlockingRecv error: recv got EOF from 127.0.98.3:55013 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 00:10:05.519730 9521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:06.273414 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:06.257339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:44549)
0924 00:10:06.257481 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:06.257492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:06.257548 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:06.257689 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:06.257689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:06.257849 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:06.257938 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.257939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.258586 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.258587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.259151 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.259153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.259525 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.259526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.259596 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.271545 (+ 11949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:06.271551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:06.273346 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260924 00:10:06.273624 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:06.257402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44549 (local address 127.0.98.62:34829)
0924 00:10:06.257549 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:06.257551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:06.257588 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:10:06.257608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:06.257609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:06.257618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:06.257668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:06.258009 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.258498 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.259242 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.259436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.259502 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.273511 (+ 14009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44549: BlockingRecv error: recv got EOF from 127.0.98.1:44549 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:06.390211 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:06.385075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:52075)
0924 00:10:06.385262 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:06.385275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:06.385383 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:06.385596 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:06.385598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:06.385854 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:06.385944 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.385947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.386642 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.386643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.387304 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.387307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.387697 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.387698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.387750 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.388242 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:06.388247 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:06.390104 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
W20260924 00:10:06.390251 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:06.385177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52075 (local address 127.0.98.62:34829)
0924 00:10:06.385255 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:06.385257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:06.385363 (+ 106us) server_negotiation.cc:407] Connection header received
0924 00:10:06.385454 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:06.385455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:06.385488 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:06.385560 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:06.386032 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.386533 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.387382 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.387630 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.387689 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.390188 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52075: BlockingRecv error: recv got EOF from 127.0.98.2:52075 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:06 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:06.525224 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:06.520215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:34469)
0924 00:10:06.520359 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:06.520389 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:06.520481 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:06.520652 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:06.520653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:06.520883 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:06.520954 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.520958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.521736 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.521737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.522288 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:06.522291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.522792 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.522793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:06.522838 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.523340 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:06.523345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:06.525155 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
W20260924 00:10:06.525297 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:06.520240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34469 (local address 127.0.98.62:34829)
0924 00:10:06.520320 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:06.520323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:06.520459 (+ 136us) server_negotiation.cc:407] Connection header received
0924 00:10:06.520564 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:06.520564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:06.520578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:06.520635 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:06.521072 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.521641 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.522509 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:06.522710 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:06.522777 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:06.525249 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34469: BlockingRecv error: recv got EOF from 127.0.98.3:34469 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:07.278930 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:07.273949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:56493)
0924 00:10:07.274272 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:07.274282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:07.274350 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:07.274500 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:07.274501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:07.274693 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:07.274742 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.274744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.275483 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.275484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.276106 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.276109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.276429 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.276429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.276467 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.277012 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:07.277016 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:07.278830 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":239,"threads_started":1}
W20260924 00:10:07.278983 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:07.274062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56493 (local address 127.0.98.62:34829)
0924 00:10:07.274269 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:07.274271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:07.274351 (+ 80us) server_negotiation.cc:407] Connection header received
0924 00:10:07.274402 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:07.274403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:07.274414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:07.274464 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:07.274824 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.275397 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.276180 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.276367 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.276420 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.278911 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56493: BlockingRecv error: recv got EOF from 127.0.98.1:56493 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":131,"threads_started":1}
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:07.395467 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:07.390762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:46529)
0924 00:10:07.390948 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:07.390962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:07.391052 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:07.391195 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:07.391196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:07.391393 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:07.391439 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.391441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.392136 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.392137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.392654 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.392656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.393025 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.393026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.393077 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.393596 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:07.393600 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:07.395358 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
W20260924 00:10:07.395519 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:07.390871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46529 (local address 127.0.98.62:34829)
0924 00:10:07.390952 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:07.390955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:07.391037 (+ 82us) server_negotiation.cc:407] Connection header received
0924 00:10:07.391109 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:07.391110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:07.391124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:07.391177 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:07.391535 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.392031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.392742 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.392939 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.393025 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.395454 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46529: BlockingRecv error: recv got EOF from 127.0.98.2:46529 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:07 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:07.530886 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:07.525853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:52073)
0924 00:10:07.526022 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:07.526035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:07.526105 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:07.526342 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:07.526343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:07.526527 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:07.526582 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.526584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.527320 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.527321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.527861 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:07.527863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.528210 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.528211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:07.528250 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.528784 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:07.528789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:07.530827 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
W20260924 00:10:07.531002 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:07.526099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52073 (local address 127.0.98.62:34829)
0924 00:10:07.526191 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:07.526193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:07.526202 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:07.526226 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:07.526227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:07.526239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:07.526296 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:07.526678 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.527171 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.527955 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:07.528142 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:07.528200 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:07.530925 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52073: BlockingRecv error: recv got EOF from 127.0.98.3:52073 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:08.284106 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:08.279472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:50785)
0924 00:10:08.279610 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:08.279623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:08.279685 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:08.279823 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:08.279823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:08.279982 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:08.280031 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.280032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.280721 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.280722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.281249 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.281250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.281579 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.281580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.281618 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.282042 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:08.282047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:08.284048 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260924 00:10:08.284194 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:08.279551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50785 (local address 127.0.98.62:34829)
0924 00:10:08.279688 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:08.279690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:08.279705 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:10:08.279730 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:08.279731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:08.279740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:08.279806 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:08.280099 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.280646 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.281322 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.281511 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.281597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.284138 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50785: BlockingRecv error: recv got EOF from 127.0.98.1:50785 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:08.400857 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:08.395993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:35047)
0924 00:10:08.396206 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:08.396218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:08.396310 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:08.396452 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:08.396453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:08.396632 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:08.396678 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.396680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.397352 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.397353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.397886 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.397887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.398270 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.398272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.398328 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.398916 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:08.398921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:08.400799 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
W20260924 00:10:08.400990 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:08.396085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35047 (local address 127.0.98.62:34829)
0924 00:10:08.396188 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:08.396191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:08.396291 (+ 100us) server_negotiation.cc:407] Connection header received
0924 00:10:08.396372 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:08.396373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:08.396385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:08.396433 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:08.396759 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.397262 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.397991 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.398182 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.398269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.400908 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35047: BlockingRecv error: recv got EOF from 127.0.98.2:35047 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:08 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:08.536361 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:08.531470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:60199)
0924 00:10:08.531658 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:08.531670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:08.531762 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:08.531931 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:08.531932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:08.532119 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:08.532186 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.532188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.532892 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.532894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.533434 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:08.533435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.533817 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.533818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:08.533873 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.534400 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:08.534406 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:08.536257 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":138,"threads_started":1}
W20260924 00:10:08.536399 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:08.531573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60199 (local address 127.0.98.62:34829)
0924 00:10:08.531646 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:08.531649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:08.531743 (+ 94us) server_negotiation.cc:407] Connection header received
0924 00:10:08.531837 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:08.531838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:08.531852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:08.531923 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:08.532276 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.532790 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.533547 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:08.533742 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:08.533819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:08.536334 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60199: BlockingRecv error: recv got EOF from 127.0.98.3:60199 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:09.289496 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:09.284679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:41965)
0924 00:10:09.284801 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:09.284815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:09.284887 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:09.285069 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:09.285070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:09.285235 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:09.285283 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.285291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.285979 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.285980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.286515 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.286516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.287000 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.287001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.287043 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.287544 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:09.287549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:09.289439 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260924 00:10:09.289672 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:09.284754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41965 (local address 127.0.98.62:34829)
0924 00:10:09.284891 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:09.284893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:09.284920 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:10:09.284948 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:09.284949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:09.284966 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:09.285018 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:09.285390 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.285899 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.286644 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.286929 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.286992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.289573 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41965: BlockingRecv error: recv got EOF from 127.0.98.1:41965 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:09.405803 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:09.401315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:54353)
0924 00:10:09.401514 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:09.401525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:09.401582 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:09.401725 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:09.401725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:09.401888 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:09.401936 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.401938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.402570 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.402571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.403098 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.403099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.403558 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.403559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.403604 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.404043 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:09.404047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:09.405744 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
W20260924 00:10:09.405946 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:09.401416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54353 (local address 127.0.98.62:34829)
0924 00:10:09.401473 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:09.401475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:09.401568 (+ 93us) server_negotiation.cc:407] Connection header received
0924 00:10:09.401638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:09.401639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:09.401649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:09.401708 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:09.402006 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.402496 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.403174 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.403479 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.403541 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.405860 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54353: BlockingRecv error: recv got EOF from 127.0.98.2:54353 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:09 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:09.541517 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:09.536898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:55609)
0924 00:10:09.537046 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:09.537057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:09.537134 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:09.537300 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:09.537300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:09.537463 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:09.537512 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.537514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.538196 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.538197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.538751 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:09.538753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.539079 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.539080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:09.539120 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.539716 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:09.539721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:09.541441 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260924 00:10:09.541682 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:09.537020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55609 (local address 127.0.98.62:34829)
0924 00:10:09.537153 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:09.537156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:09.537168 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:10:09.537201 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:09.537202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:09.537215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:09.537278 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:09.537617 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.538118 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.538819 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:09.539002 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:09.539067 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:09.541583 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55609: BlockingRecv error: recv got EOF from 127.0.98.3:55609 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:10.294927 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:10.290009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:45819)
0924 00:10:10.290157 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:10.290172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:10.290242 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:10.290377 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:10.290378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:10.290545 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:10.290635 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.290638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.291327 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.291328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.291887 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.291889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.292365 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.292367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.292430 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.293035 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:10.293041 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:10.294861 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260924 00:10:10.295115 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:10.290080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45819 (local address 127.0.98.62:34829)
0924 00:10:10.290253 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:10.290255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:10.290264 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:10.290289 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:10.290289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:10.290300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:10.290363 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:10.290715 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.291256 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.291966 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.292269 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.292345 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.294985 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45819: BlockingRecv error: recv got EOF from 127.0.98.1:45819 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:10.410931 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:10.406357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:48039)
0924 00:10:10.406542 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:10.406557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:10.406636 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:10.406801 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:10.406802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:10.406980 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:10.407043 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.407045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.407773 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.407774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.408302 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.408303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.408626 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.408627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.408669 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.409143 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:10.409147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:10.410873 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
W20260924 00:10:10.411036 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:10.406614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48039 (local address 127.0.98.62:34829)
0924 00:10:10.406680 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:10.406682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:10.406689 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:10:10.406714 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:10.406715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:10.406725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:10.406777 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:10.407111 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.407697 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.408374 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.408555 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.408607 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.410970 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48039: BlockingRecv error: recv got EOF from 127.0.98.2:48039 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:10 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:10.547019 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:10.542170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:34109)
0924 00:10:10.542321 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:10.542340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:10.542393 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:10.542541 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:10.542542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:10.542700 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:10.542749 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.542750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.543465 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.543466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.544011 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:10.544013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.544346 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.544347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:10.544385 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.544879 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:10.544884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:10.546939 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260924 00:10:10.547108 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:10.542285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34109 (local address 127.0.98.62:34829)
0924 00:10:10.542386 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:10.542388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:10.542400 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:10:10.542444 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:10.542445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:10.542458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:10.542524 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:10.542818 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.543391 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.544102 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:10.544282 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:10.544332 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:10.547058 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34109: BlockingRecv error: recv got EOF from 127.0.98.3:34109 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:11.312386 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:11.295597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:35245)
0924 00:10:11.295763 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:11.295775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:11.295838 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:11.295977 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:11.295978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:11.296144 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:11.296192 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.296194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.296888 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.296889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.297434 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.297436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.297878 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.297879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.297920 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.310292 (+ 12372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:11.310301 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:11.312313 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
W20260924 00:10:11.312541 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:11.295713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35245 (local address 127.0.98.62:34829)
0924 00:10:11.295849 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:11.295851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:11.295860 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:11.295884 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:11.295885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:11.295896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:11.295958 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:11.296268 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.296810 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.297554 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.297802 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.297865 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.312450 (+ 14585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35245: BlockingRecv error: recv got EOF from 127.0.98.1:35245 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:11.416193 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:11.411508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:39261)
0924 00:10:11.411714 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:11.411725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:11.411785 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:11.411916 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:11.411917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:11.412086 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:11.412133 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.412134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.412778 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.412778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.413301 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.413302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.413611 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.413612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.413649 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.414180 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:11.414185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:11.416105 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
W20260924 00:10:11.416245 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:11.411594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39261 (local address 127.0.98.62:34829)
0924 00:10:11.411661 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:11.411663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:11.411781 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:10:11.411832 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:11.411832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:11.411844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:11.411897 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:11.412195 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.412707 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.413371 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.413550 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.413630 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.416198 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39261: BlockingRecv error: recv got EOF from 127.0.98.2:39261 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:11 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:11.552858 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:11.547713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:44091)
0924 00:10:11.547940 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:11.547955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:11.548041 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:11.548276 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:11.548277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:11.548460 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:11.548506 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.548507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.549234 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.549236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.549759 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:11.549761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.550168 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.550169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:11.550225 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.550772 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:11.550778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:11.552800 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":130,"threads_started":1}
W20260924 00:10:11.552970 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:11.547866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44091 (local address 127.0.98.62:34829)
0924 00:10:11.547945 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:11.547948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:11.548011 (+ 63us) server_negotiation.cc:407] Connection header received
0924 00:10:11.548137 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:11.548138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:11.548153 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:11.548233 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:11.548593 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.549152 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.549872 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:11.550082 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:11.550170 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:11.552894 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44091: BlockingRecv error: recv got EOF from 127.0.98.3:44091 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:12.318092 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:12.312914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:49121)
0924 00:10:12.313046 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:12.313060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:12.313142 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:12.313293 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:12.313294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:12.313456 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:12.313505 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.313507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.314359 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.314360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.314889 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.314891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.315313 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.315313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.315356 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.315916 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:12.315921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:12.318032 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260924 00:10:12.318195 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:12.313009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49121 (local address 127.0.98.62:34829)
0924 00:10:12.313163 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:12.313165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:12.313174 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:12.313197 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:12.313198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:12.313208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:12.313270 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:12.313574 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.314277 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.314966 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.315245 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.315314 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.318116 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49121: BlockingRecv error: recv got EOF from 127.0.98.1:49121 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:12.421164 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:12.416660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:35505)
0924 00:10:12.416831 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:12.416843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:12.416904 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:12.417028 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:12.417028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:12.417210 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:12.417271 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.417274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.417928 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.417929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.418453 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.418455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.418799 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.418800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.418838 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.419385 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:12.419390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:12.421103 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
W20260924 00:10:12.421306 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:12.416722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35505 (local address 127.0.98.62:34829)
0924 00:10:12.416782 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:12.416784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:12.416907 (+ 123us) server_negotiation.cc:407] Connection header received
0924 00:10:12.416946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:12.416947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:12.416958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:12.417009 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:12.417343 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.417832 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.418526 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.418732 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.418785 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.421224 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35505: BlockingRecv error: recv got EOF from 127.0.98.2:35505 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:12 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:12.558135 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:12.553376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:53703)
0924 00:10:12.553521 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:12.553539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:12.553589 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:12.553760 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:12.553760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:12.553922 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:12.553971 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.553972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.554612 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.554643 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.555175 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:12.555177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.555666 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.555667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:12.555715 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.556190 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:12.556195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:12.558034 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260924 00:10:12.558187 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:12.553523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53703 (local address 127.0.98.62:34829)
0924 00:10:12.553593 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:12.553595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:12.553612 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:10:12.553668 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:12.553668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:12.553680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:12.553741 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:12.554045 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.554534 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.555339 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:12.555535 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:12.555591 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:12.558120 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53703: BlockingRecv error: recv got EOF from 127.0.98.3:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:13.323168 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:13.318567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:35705)
0924 00:10:13.318711 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:13.318726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:13.318808 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:13.318954 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:13.318955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:13.319117 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:13.319165 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.319167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.319924 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.319929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.320456 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.320457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.320776 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.320776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.320830 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.321283 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:13.321287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:13.323107 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
W20260924 00:10:13.323330 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:13.318700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35705 (local address 127.0.98.62:34829)
0924 00:10:13.318829 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:13.318831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:13.318840 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:10:13.318864 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:13.318865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:13.318875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:13.318934 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:13.319325 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.319815 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.320529 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.320710 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.320764 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.323198 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35705: BlockingRecv error: recv got EOF from 127.0.98.1:35705 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:13.426672 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:13.421684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:44959)
0924 00:10:13.421874 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:13.421893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:13.421980 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:13.422163 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:13.422164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:13.422345 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:13.422388 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.422390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.423346 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.423347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.423869 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.423871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.424245 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.424246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.424323 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.424805 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:13.424810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:13.426615 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
W20260924 00:10:13.426769 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:13.421754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44959 (local address 127.0.98.62:34829)
0924 00:10:13.421834 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:13.421837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:13.421958 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:10:13.422058 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:13.422059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:13.422078 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:13.422138 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:13.422475 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.423189 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.423955 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.424160 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.424245 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.426721 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44959: BlockingRecv error: recv got EOF from 127.0.98.2:44959 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:13 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:13.563274 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:13.558688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:42453)
0924 00:10:13.558843 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:13.558863 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:13.558920 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:13.559065 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:13.559066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:13.559264 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:13.559312 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.559313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.559965 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.559966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.560474 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:13.560475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.560808 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.560809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:13.560845 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.561306 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:13.561311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:13.563195 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
W20260924 00:10:13.563442 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:13.558806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42453 (local address 127.0.98.62:34829)
0924 00:10:13.558859 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:13.558861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:13.558907 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:10:13.558989 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:13.558989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:13.559001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:13.559045 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:13.559384 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.559896 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.560547 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:13.560743 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:13.560795 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:13.563358 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42453: BlockingRecv error: recv got EOF from 127.0.98.3:42453 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:14.328516 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:14.323724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:57561)
0924 00:10:14.323900 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:14.323915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:14.323985 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:14.324130 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:14.324131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:14.324311 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:14.324359 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.324360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.325059 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.325060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.325588 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.325589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.325952 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.325953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.325993 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.326467 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:14.326471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:14.328458 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
W20260924 00:10:14.328665 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:14.323800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57561 (local address 127.0.98.62:34829)
0924 00:10:14.323954 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:14.323956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:14.323970 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:10:14.324039 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:14.324040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:14.324050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:14.324111 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:14.324425 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.324983 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.325681 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.325883 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.325942 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.328575 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57561: BlockingRecv error: recv got EOF from 127.0.98.1:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:14.431915 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:14.427221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:44983)
0924 00:10:14.427427 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:14.427438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:14.427507 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:14.427640 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:14.427641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:14.427806 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:14.427856 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.427858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.428512 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.428512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.429053 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.429055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.429401 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.429402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.429443 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.429923 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:14.429927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:14.431856 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260924 00:10:14.432081 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:14.427313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44983 (local address 127.0.98.62:34829)
0924 00:10:14.427382 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:14.427384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:14.427507 (+ 123us) server_negotiation.cc:407] Connection header received
0924 00:10:14.427556 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:14.427557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:14.427567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:14.427618 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:14.427924 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.428435 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.429137 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.429333 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.429385 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.432010 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44983: BlockingRecv error: recv got EOF from 127.0.98.2:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:14 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:14.568457 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:14.563735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:48331)
0924 00:10:14.563887 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:14.563898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:14.563960 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:14.564108 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:14.564108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:14.564293 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:14.564344 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.564346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.564990 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.564990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.565520 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:14.565521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.565849 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.565850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:14.565888 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.566346 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:14.566350 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:14.568361 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260924 00:10:14.568501 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:14.563834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48331 (local address 127.0.98.62:34829)
0924 00:10:14.563905 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:14.563907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:14.563961 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:10:14.564028 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:14.564029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:14.564041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:14.564087 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:14.564418 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.564921 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.565593 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:14.565785 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:14.565836 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:14.568439 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48331: BlockingRecv error: recv got EOF from 127.0.98.3:48331 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:15.333705 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:15.329045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:50915)
0924 00:10:15.329188 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:15.329199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:15.329270 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:15.329404 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:15.329405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:15.329570 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:15.329616 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.329617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.330250 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.330251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.330754 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.330755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.331359 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.331376 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.331439 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.331928 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:15.331933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:15.333636 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
W20260924 00:10:15.333849 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:15.329158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50915 (local address 127.0.98.62:34829)
0924 00:10:15.329279 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:15.329281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:15.329291 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:10:15.329316 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:15.329316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:15.329326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:15.329387 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:15.329679 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.330169 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.330875 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.331208 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.331315 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.333755 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50915: BlockingRecv error: recv got EOF from 127.0.98.1:50915 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:15.437510 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:15.432389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:52951)
0924 00:10:15.432568 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:15.432581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:15.432676 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:15.432819 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:15.432820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:15.433011 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:15.433055 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.433057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.433769 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.433770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.434291 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.434292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.434667 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.434668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.434724 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.435322 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:15.435326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:15.437415 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
W20260924 00:10:15.437565 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:15.432484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52951 (local address 127.0.98.62:34829)
0924 00:10:15.432563 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:15.432565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:15.432656 (+ 91us) server_negotiation.cc:407] Connection header received
0924 00:10:15.432745 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:15.432746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:15.432761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:15.432813 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:15.433142 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.433676 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.434382 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.434581 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.434666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.437503 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52951: BlockingRecv error: recv got EOF from 127.0.98.2:52951 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:15 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:15.573664 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:15.569002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:38087)
0924 00:10:15.569179 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:15.569190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:15.569277 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:15.569420 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:15.569421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:15.569601 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:15.569649 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.569650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.570327 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.570328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.570853 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:15.570855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.571258 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.571259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:15.571316 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.571824 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:15.571828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:15.573603 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
W20260924 00:10:15.573817 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:15.569099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38087 (local address 127.0.98.62:34829)
0924 00:10:15.569156 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:15.569159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:15.569234 (+ 75us) server_negotiation.cc:407] Connection header received
0924 00:10:15.569331 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:15.569332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:15.569346 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:15.569402 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:15.569731 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.570231 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.570938 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:15.571133 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:15.571243 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:15.573713 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38087: BlockingRecv error: recv got EOF from 127.0.98.3:38087 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:16.344702 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:16.334161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:40793)
0924 00:10:16.334308 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:16.334319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:16.334371 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:16.334491 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:16.334491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:16.334653 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:16.334710 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.334711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.335399 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.335399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.335939 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.335940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.336263 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.336264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.336302 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.342784 (+ 6482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:16.342789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:16.344628 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
W20260924 00:10:16.344836 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:16.334272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40793 (local address 127.0.98.62:34829)
0924 00:10:16.334380 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:16.334382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:16.334390 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:16.334425 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:16.334426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:16.334436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:16.334480 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:16.334774 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.335330 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.336014 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.336201 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.336252 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.344747 (+ 8495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40793: BlockingRecv error: recv got EOF from 127.0.98.1:40793 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:16.442965 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:16.438126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:33635)
0924 00:10:16.438277 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:16.438291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:16.438362 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:16.438501 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:16.438502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:16.438663 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:16.438713 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.438714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.439680 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.439681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.440239 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.440241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.440571 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.440572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.440611 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.441166 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:16.441171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:16.442893 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260924 00:10:16.443092 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:16.438252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33635 (local address 127.0.98.62:34829)
0924 00:10:16.438329 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:16.438331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:16.438343 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:10:16.438425 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:16.438426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:16.438437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:16.438479 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:16.438818 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.439625 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.440323 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.440505 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.440557 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.442993 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33635: BlockingRecv error: recv got EOF from 127.0.98.2:33635 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:16 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:16.578701 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:16.574156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:52281)
0924 00:10:16.574309 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:16.574320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:16.574376 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:16.574515 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:16.574516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:16.574692 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:16.574742 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.574743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.575459 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.575460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.575990 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:16.575992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.576322 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.576322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:16.576361 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.576819 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:16.576823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:16.578642 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260924 00:10:16.578860 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:16.574246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52281 (local address 127.0.98.62:34829)
0924 00:10:16.574330 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:16.574332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:16.574369 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:10:16.574432 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:16.574433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:16.574444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:16.574485 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:16.574811 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.575368 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.576067 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:16.576255 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:16.576309 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:16.578767 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52281: BlockingRecv error: recv got EOF from 127.0.98.3:52281 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:17.349881 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:17.345272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:35405)
0924 00:10:17.345432 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:17.345451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:17.345505 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:17.345627 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:17.345627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:17.345793 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:17.345840 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.345841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.346505 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.346506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.347040 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.347042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.347462 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.347463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.347520 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.347969 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:17.347974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:17.349824 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260924 00:10:17.350023 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:17.345308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35405 (local address 127.0.98.62:34829)
0924 00:10:17.345453 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:17.345455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:17.345509 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:10:17.345550 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:17.345551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:17.345561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:17.345622 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:17.345900 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.346419 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.347119 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.347352 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.347410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.349931 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35405: BlockingRecv error: recv got EOF from 127.0.98.1:35405 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:17.447899 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:17.443451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:52201)
0924 00:10:17.443598 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:17.443607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:17.443664 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:17.443792 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:17.443793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:17.443948 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:17.443997 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.443999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.444647 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.444647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.445183 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.445184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.445497 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.445497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.445536 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.445957 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:17.445961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:17.447842 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
W20260924 00:10:17.448021 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:17.443511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52201 (local address 127.0.98.62:34829)
0924 00:10:17.443580 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:17.443583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:17.443666 (+ 83us) server_negotiation.cc:407] Connection header received
0924 00:10:17.443710 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:17.443711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:17.443722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:17.443786 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:17.444064 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.444574 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.445255 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.445435 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.445484 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.447943 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52201: BlockingRecv error: recv got EOF from 127.0.98.2:52201 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:17 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:17.584034 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:17.579258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:34507)
0924 00:10:17.579407 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:17.579418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:17.579480 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:17.579626 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:17.579626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:17.579821 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:17.579871 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.579873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.580503 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.580504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.581028 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:17.581030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.581344 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.581345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:17.581385 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.581993 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:17.581999 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:17.583965 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260924 00:10:17.584160 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:17.579322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34507 (local address 127.0.98.62:34829)
0924 00:10:17.579380 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:17.579382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:17.579484 (+ 102us) server_negotiation.cc:407] Connection header received
0924 00:10:17.579546 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:17.579547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:17.579559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:17.579603 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:17.579941 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.580430 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.581104 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:17.581281 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:17.581332 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:17.584057 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34507: BlockingRecv error: recv got EOF from 127.0.98.3:34507 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:18.354887 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:18.350415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:55913)
0924 00:10:18.350554 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:18.350566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:18.350625 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:18.350789 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:18.350790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:18.350943 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:18.350992 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.350993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.351708 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.351708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.352236 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.352238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.352554 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.352555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.352593 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.353010 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:18.353014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:18.354809 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260924 00:10:18.354982 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:18.350538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55913 (local address 127.0.98.62:34829)
0924 00:10:18.350656 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:18.350658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:18.350674 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:10:18.350697 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:18.350698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:18.350708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:18.350762 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:18.351052 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.351640 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.352310 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.352493 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.352542 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.354896 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55913: BlockingRecv error: recv got EOF from 127.0.98.1:55913 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:18.452927 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:18.448373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:38301)
0924 00:10:18.448520 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:18.448531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:18.448593 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:18.448719 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:18.448719 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:18.448940 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:18.448988 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.448990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.449638 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.449639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.450171 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.450173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.450493 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.450493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.450531 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.451053 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:18.451057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:18.452829 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
W20260924 00:10:18.452978 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:18.448484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38301 (local address 127.0.98.62:34829)
0924 00:10:18.448527 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:18.448529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:18.448577 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:10:18.448638 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:18.448639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:18.448649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:18.448700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:18.449054 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.449550 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.450241 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.450431 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.450481 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.452914 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38301: BlockingRecv error: recv got EOF from 127.0.98.2:38301 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:18 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:18.589581 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:18.584746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:33765)
0924 00:10:18.584919 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:18.584932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:18.585006 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:18.585172 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:18.585173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:18.585388 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:18.585449 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.585451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.586162 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.586163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.586684 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:18.586685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.587122 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.587123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:18.587166 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.587744 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:18.587749 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:18.589520 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
W20260924 00:10:18.589748 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:18.584832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33765 (local address 127.0.98.62:34829)
0924 00:10:18.584901 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:18.584905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:18.585014 (+ 109us) server_negotiation.cc:407] Connection header received
0924 00:10:18.585068 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:18.585069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:18.585086 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:18.585152 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:18.585541 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.586076 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.586791 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:18.587052 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:18.587112 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:18.589657 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33765: BlockingRecv error: recv got EOF from 127.0.98.3:33765 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:19.360200 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:19.355661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:50153)
0924 00:10:19.355789 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:19.355805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:19.355854 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:19.356015 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:19.356017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:19.356205 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:19.356255 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.356256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.356964 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.356965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.357503 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.357505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.357900 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.357901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.357965 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.358423 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:19.358428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:19.360142 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260924 00:10:19.360344 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:19.355737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50153 (local address 127.0.98.62:34829)
0924 00:10:19.355866 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:19.355868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:19.355876 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:19.355900 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:19.355900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:19.355910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:19.355976 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:19.356356 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.356868 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.357595 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.357809 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.357896 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.360255 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50153: BlockingRecv error: recv got EOF from 127.0.98.1:50153 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:19.458155 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:19.453428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:60055)
0924 00:10:19.453601 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:19.453612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:19.453691 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:19.453845 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:19.453846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:19.454015 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:19.454062 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.454064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.454724 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.454725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.455282 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.455284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.455678 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.455679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.455716 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.456182 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:19.456186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:19.458094 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
W20260924 00:10:19.458334 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:19.453491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60055 (local address 127.0.98.62:34829)
0924 00:10:19.453540 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:19.453543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:19.453676 (+ 133us) server_negotiation.cc:407] Connection header received
0924 00:10:19.453755 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:19.453756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:19.453768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:19.453830 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:19.454138 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.454646 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.455386 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.455592 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.455666 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.458226 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60055: BlockingRecv error: recv got EOF from 127.0.98.2:60055 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:19 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:19.594731 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:19.590067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:54989)
0924 00:10:19.590268 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:19.590281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:19.590343 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:19.590490 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:19.590491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:19.590652 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:19.590702 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.590703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.591373 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.591373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.591912 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:19.591915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.592235 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.592236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:19.592338 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.592879 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:19.592884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:19.594641 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
W20260924 00:10:19.594887 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:19.590161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54989 (local address 127.0.98.62:34829)
0924 00:10:19.590229 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:19.590231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:19.590331 (+ 100us) server_negotiation.cc:407] Connection header received
0924 00:10:19.590406 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:19.590407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:19.590419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:19.590463 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:19.590768 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.591281 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.591991 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:19.592169 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:19.592220 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:19.594810 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54989: BlockingRecv error: recv got EOF from 127.0.98.3:54989 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:10:20 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:20 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:20.365373 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.360804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:43049)
0924 00:10:20.360942 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:20.360953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:20.361000 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:20.361160 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:20.361161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:20.361345 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:20.361396 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.361398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.362047 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.362048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.362591 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.362592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.362939 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.362940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.362983 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.363556 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:20.363561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:20.365301 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260924 00:10:20.365476 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:20.360884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43049 (local address 127.0.98.62:34829)
0924 00:10:20.361030 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:20.361032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:20.361040 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:10:20.361061 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:20.361061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:20.361071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:20.361133 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:20.361470 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.361974 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.362673 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.362872 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.362924 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.365401 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43049: BlockingRecv error: recv got EOF from 127.0.98.1:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
Sep 24 00:10:20 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:10:20 dist-test-slave-3q8k krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:10:20.463186 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.458711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:39711)
0924 00:10:20.458887 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:20.458906 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:20.458969 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:20.459084 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:20.459084 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:20.459266 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:20.459331 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.459333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.460010 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.460011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.460538 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.460539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.460899 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.460900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.460939 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.461473 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:20.461478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:20.463116 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
W20260924 00:10:20.463326 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:20.458841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39711 (local address 127.0.98.62:34829)
0924 00:10:20.458912 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:20.458914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:20.458954 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:10:20.459016 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:20.459017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:20.459027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:20.459072 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:20.459391 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.459937 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.460614 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.460834 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.460888 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.463262 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39711: BlockingRecv error: recv got EOF from 127.0.98.2:39711 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:10:20.509167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9108
I20260924 00:10:20.513681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9259
I20260924 00:10:20.518041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9392
I20260924 00:10:20.521976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9037
2026-09-24T00:10:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17556 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T00:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:20Z Disabled control of system clock
I20260924 00:10:20.549665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38555
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:10:20.629913 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:20.630129 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:20.630149 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:20.631281 9588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:20.631318 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:20.631330 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:20.631343 9588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:20.631356 9588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:20.632633 9588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:20.632825 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:20.633067 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:20.638397 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:20.638445 9588 server_base.cc:1034] running on GCE node
W20260924 00:10:20.638424 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:20.638393 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:20.638865 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:20.639117 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:20.640269 9588 hybrid_clock.cc:648] HybridClock initialized: now 1790208620640272 us; error 46 us; skew 500 ppm
I20260924 00:10:20.641650 9588 webserver.cc:460] Webserver started at http://127.0.98.62:37425/ using document root <none> and password file <none>
I20260924 00:10:20.641871 9588 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:20.641914 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:20.642005 9588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:20.643162 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "ac3f2363b1de4165a7f302f9e4b7eca8"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "b38046d62be313648a1784c451c0035f"
server_key_iv: "05825e7dbd05bc8a437144cd14ba6049"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.643612 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac3f2363b1de4165a7f302f9e4b7eca8"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "b38046d62be313648a1784c451c0035f"
server_key_iv: "05825e7dbd05bc8a437144cd14ba6049"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.644903 9588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:20.645681 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.645900 9588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:20.646018 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "ac3f2363b1de4165a7f302f9e4b7eca8"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "b38046d62be313648a1784c451c0035f"
server_key_iv: "05825e7dbd05bc8a437144cd14ba6049"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.646095 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:20.655280 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:20.655622 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:20.655776 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:20.659737 9588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38555
I20260924 00:10:20.659771 9654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38555 every 8 connection(s)
I20260924 00:10:20.660161 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:20.660907 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:20.663744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9588
I20260924 00:10:20.663754 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8: Bootstrap starting.
I20260924 00:10:20.663848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:10:20.664045 392 external_mini_cluster.cc:1428] Setting key 99aa6cfc01c9394ea03daeee7bea2975
I20260924 00:10:20.664404 9655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:20.664983 9655 log.cc:824] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:20.665761 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8: No bootstrap required, opened a new log
I20260924 00:10:20.667820 9655 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } }
I20260924 00:10:20.668015 9655 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:20.668066 9655 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3f2363b1de4165a7f302f9e4b7eca8, State: Initialized, Role: FOLLOWER
I20260924 00:10:20.668201 9655 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } }
I20260924 00:10:20.668316 9655 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:20.668360 9655 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:20.668409 9655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:20.669176 9655 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } }
I20260924 00:10:20.669296 9655 leader_election.cc:302] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3f2363b1de4165a7f302f9e4b7eca8; no voters:
I20260924 00:10:20.669466 9655 leader_election.cc:288] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:10:20.669493 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:20.669601 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [term 1 LEADER]: Becoming Leader. State: Replica: ac3f2363b1de4165a7f302f9e4b7eca8, State: Running, Role: LEADER
I20260924 00:10:20.669682 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } }
I20260924 00:10:20.669746 9655 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:20.669977 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } } }
I20260924 00:10:20.670043 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:20.670208 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3f2363b1de4165a7f302f9e4b7eca8. Latest consensus state: current_term: 1 leader_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3f2363b1de4165a7f302f9e4b7eca8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38555 } } }
I20260924 00:10:20.670315 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:20.670295 9665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:20.670612 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.664591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.98.62:38555)
0924 00:10:20.664798 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:20.664805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:20.664831 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:10:20.665319 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:20.665326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:20.665384 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:20.665499 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:20.665746 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.666727 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.667914 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.668490 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.668603 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.669295 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:20.669300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:20.669300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:20.669546 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:20.669596 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:20.669611 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:20.669987 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:20.670025 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 00:10:20.670137 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260924 00:10:20.670943 9665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:20.672760 9665 catalog_manager.cc:1347] Generated new cluster ID: 52d1798e93aa435eb09a39169219703d
I20260924 00:10:20.672814 9665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:10:20.687250 9665 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:10:20.687839 9665 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:20.691738 9665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3f2363b1de4165a7f302f9e4b7eca8: Generated new TSK 0
I20260924 00:10:20.691937 9665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:10:20.708426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 00:10:20.788743 9679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:20.788954 9679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:20.788975 9679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:20.790122 9679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:20.790167 9679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:20.790192 9679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:20.791527 9679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:20.791785 9679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:20.792086 9679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:20.797677 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:20.797720 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:20.797683 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:20.797995 9679 server_base.cc:1034] running on GCE node
I20260924 00:10:20.798210 9679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:20.798475 9679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:20.799623 9679 hybrid_clock.cc:648] HybridClock initialized: now 1790208620799610 us; error 28 us; skew 500 ppm
I20260924 00:10:20.800830 9679 webserver.cc:460] Webserver started at http://127.0.98.1:39139/ using document root <none> and password file <none>
I20260924 00:10:20.801085 9679 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:20.801133 9679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:20.801254 9679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:20.802302 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "9667693dc8c84a73bfa78bb69036f86e"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "4cc3f09a3413cc072be97c98a6c7e5b6"
server_key_iv: "559f0f9affae842e97cdfb7db4f4b375"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.802652 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9667693dc8c84a73bfa78bb69036f86e"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "4cc3f09a3413cc072be97c98a6c7e5b6"
server_key_iv: "559f0f9affae842e97cdfb7db4f4b375"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.804018 9679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:20.804776 9693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.804983 9679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:10:20.805087 9679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "9667693dc8c84a73bfa78bb69036f86e"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "4cc3f09a3413cc072be97c98a6c7e5b6"
server_key_iv: "559f0f9affae842e97cdfb7db4f4b375"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.805160 9679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:20.830152 9679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:20.830500 9679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:20.830655 9679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:20.830904 9679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:20.831288 9679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:20.831342 9679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.831384 9679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:20.831411 9679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.837397 9679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46835
I20260924 00:10:20.837440 9806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46835 every 8 connection(s)
I20260924 00:10:20.837770 9679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:10:20.843011 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.838537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36617 (local address 127.0.98.62:38555)
0924 00:10:20.838609 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:20.838611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:20.838868 (+ 257us) server_negotiation.cc:407] Connection header received
0924 00:10:20.839428 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:20.839429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:20.839444 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:20.839510 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:20.840167 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.840676 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.841660 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.841876 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.841962 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.842738 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:20.842740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:20.842740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:20.842745 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:20.842759 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:20.842763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:20.842811 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:20.842908 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 00:10:20.842932 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:10:20.843286 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.838522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38555 (local address 127.0.98.1:36617)
0924 00:10:20.838724 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:20.838759 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:20.839316 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:20.839586 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:20.839589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:20.839597 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:20.840073 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.840077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.840774 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.840775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.841557 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.841559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.841973 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.841974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.842101 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.842602 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:20.842602 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:20.842631 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:20.842652 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:20.842816 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:20.842821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:20.842824 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:10:20.842909 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:10:20.843008 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260924 00:10:20.843791 9807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38555
I20260924 00:10:20.843892 9807 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:20.843940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9679
I20260924 00:10:20.844017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:10:20.844091 9807 heartbeater.cc:511] Master 127.0.98.62:38555 requested a full tablet report, sending...
I20260924 00:10:20.844221 392 external_mini_cluster.cc:1428] Setting key 66e9dab01e39e62d01c356b28cedcf9c
I20260924 00:10:20.844627 9619 ts_manager.cc:194] Registered new tserver with Master: 9667693dc8c84a73bfa78bb69036f86e (127.0.98.1:46835)
I20260924 00:10:20.845252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 00:10:20.845366 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36617
W20260924 00:10:20.930426 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:20.930636 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:20.930658 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:20.931926 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:20.931972 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:20.931998 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:20.933334 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:20.933588 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:20.933899 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:20.939033 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:20.939141 9810 server_base.cc:1034] running on GCE node
W20260924 00:10:20.939029 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:20.939038 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:20.939546 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:20.939872 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:20.941031 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790208620941014 us; error 39 us; skew 500 ppm
I20260924 00:10:20.942279 9810 webserver.cc:460] Webserver started at http://127.0.98.2:42861/ using document root <none> and password file <none>
I20260924 00:10:20.942560 9810 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:20.942606 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:20.942756 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:20.943922 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "747281bef50c4c22bafa491b63dabd2c"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "60ca587b152dce3eef75983528fd2289"
server_key_iv: "f331ce46762cb95d48ebaaf68eaf648c"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.944278 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "747281bef50c4c22bafa491b63dabd2c"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "60ca587b152dce3eef75983528fd2289"
server_key_iv: "f331ce46762cb95d48ebaaf68eaf648c"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.945547 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:10:20.946238 9824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.946422 9810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:20.946528 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "747281bef50c4c22bafa491b63dabd2c"
format_stamp: "Formatted at 2026-09-24 00:10:20 on dist-test-slave-3q8k"
server_key: "60ca587b152dce3eef75983528fd2289"
server_key_iv: "f331ce46762cb95d48ebaaf68eaf648c"
server_key_version: "encryptionkey@0"
I20260924 00:10:20.946611 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:20.970333 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:20.970656 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:20.970829 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:20.971071 9810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:20.971494 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:20.971554 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.971589 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:20.971630 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:20.977265 9810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45357
I20260924 00:10:20.977309 9937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45357 every 8 connection(s)
I20260924 00:10:20.977641 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:10:20.980273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9810
I20260924 00:10:20.980340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:10:20.980533 392 external_mini_cluster.cc:1428] Setting key 4ae072513f07e414c55fb21f02d708a3
I20260924 00:10:20.981459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--builtin_ntp_servers=127.0.98.20:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 00:10:20.983333 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.978396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55781 (local address 127.0.98.62:38555)
0924 00:10:20.978468 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:20.978469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:20.978700 (+ 231us) server_negotiation.cc:407] Connection header received
0924 00:10:20.979258 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:20.979259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:20.979271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:20.979325 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:20.980004 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.980620 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.981807 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.982027 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.982104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.983023 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:20.983024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:20.983024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:20.983029 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:20.983043 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:20.983047 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:20.983094 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:20.983194 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 00:10:20.983256 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:10:20.983598 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:20.978378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38555 (local address 127.0.98.2:55781)
0924 00:10:20.978577 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:20.978611 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:20.979124 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:20.979406 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:20.979409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:20.979417 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:20.979910 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.979914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.980697 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.980698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.981707 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:20.981710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:20.982108 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:20.982109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:20.982251 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:20.982857 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:20.982857 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:20.982901 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:20.982931 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:20.983114 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:20.983118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:20.983120 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:10:20.983192 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:10:20.983346 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260924 00:10:20.984105 9938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38555
I20260924 00:10:20.984196 9938 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:20.984412 9938 heartbeater.cc:511] Master 127.0.98.62:38555 requested a full tablet report, sending...
I20260924 00:10:20.984829 9619 ts_manager.cc:194] Registered new tserver with Master: 747281bef50c4c22bafa491b63dabd2c (127.0.98.2:45357)
I20260924 00:10:20.985219 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55781
W20260924 00:10:21.063620 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:21.063829 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:21.063850 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:21.064996 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:21.065042 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:21.065068 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:21.066483 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:21.066735 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:21.067044 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:21.072178 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.072172 9949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.072347 9941 server_base.cc:1034] running on GCE node
W20260924 00:10:21.072170 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.072660 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:21.072927 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:21.074095 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790208621074081 us; error 31 us; skew 500 ppm
I20260924 00:10:21.075783 9941 webserver.cc:460] Webserver started at http://127.0.98.3:44695/ using document root <none> and password file <none>
I20260924 00:10:21.076040 9941 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:21.076110 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:21.076265 9941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:21.077370 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "cce1f64586fb4c278ae689f100aaffd1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "568c9126c14ed3d5895806f97d64e8ff"
server_key_iv: "741ab449f9b64243ab3493cb52efee58"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.077750 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cce1f64586fb4c278ae689f100aaffd1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "568c9126c14ed3d5895806f97d64e8ff"
server_key_iv: "741ab449f9b64243ab3493cb52efee58"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.079077 9941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:10:21.079972 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.080144 9941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:21.080242 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "cce1f64586fb4c278ae689f100aaffd1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "568c9126c14ed3d5895806f97d64e8ff"
server_key_iv: "741ab449f9b64243ab3493cb52efee58"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.080343 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:21.092813 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:21.093259 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:21.093426 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:21.093672 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:21.094003 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:21.094064 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.094146 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:21.094177 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.100164 9941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35471
I20260924 00:10:21.100205 10068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35471 every 8 connection(s)
I20260924 00:10:21.100596 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:10:21.105684 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.101303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37599 (local address 127.0.98.62:38555)
0924 00:10:21.101393 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.101396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.101627 (+ 231us) server_negotiation.cc:407] Connection header received
0924 00:10:21.102133 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.102134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.102144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.102211 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.102851 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:21.103409 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:21.104335 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:21.104517 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:21.104575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:21.105436 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:21.105438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:21.105438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:21.105442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:21.105456 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:21.105460 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:21.105514 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:21.105605 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:10:21.105625 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:10:21.105902 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.101274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38555 (local address 127.0.98.3:37599)
0924 00:10:21.101473 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:21.101524 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:21.102036 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:21.102265 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:21.102268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:21.102277 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:21.102765 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:21.102768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:21.103482 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:21.103483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:21.104254 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:21.104256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:21.104594 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:21.104595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:21.104697 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:21.105278 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:21.105278 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:21.105308 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:21.105330 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:21.105530 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:21.105533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:21.105535 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:10:21.105583 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:10:21.105658 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260924 00:10:21.106418 10069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38555
I20260924 00:10:21.106527 10069 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:21.106722 10069 heartbeater.cc:511] Master 127.0.98.62:38555 requested a full tablet report, sending...
I20260924 00:10:21.106753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 9941
I20260924 00:10:21.106843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:10:21.107035 392 external_mini_cluster.cc:1428] Setting key 7ca6bb0ceb64f9ffa3722cd3574ec2d5
I20260924 00:10:21.107141 9619 ts_manager.cc:194] Registered new tserver with Master: cce1f64586fb4c278ae689f100aaffd1 (127.0.98.3:35471)
I20260924 00:10:21.107530 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37599
I20260924 00:10:21.107977 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:10:21.109786 9658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:21.109355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.0.98.62:38555)
0924 00:10:21.109408 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.109410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.109529 (+ 119us) server_negotiation.cc:407] Connection header received
0924 00:10:21.109574 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.109574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.109582 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.109629 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.109720 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42068: BlockingRecv error: recv got EOF from 127.0.0.1:42068 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:10:21.109885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9679
I20260924 00:10:21.114928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9810
I20260924 00:10:21.119313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9941
I20260924 00:10:21.123472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 9588
2026-09-24T00:10:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (600 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T00:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:21Z Disabled control of system clock
I20260924 00:10:21.149572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44075
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:10:21.230969 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:21.231201 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:21.231241 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:21.232455 10080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:21.232503 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:21.232517 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:21.232528 10080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:21.232540 10080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:21.233821 10080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:21.234021 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:21.234269 10080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:21.238968 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.238968 10088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.239033 10086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.239410 10080 server_base.cc:1034] running on GCE node
I20260924 00:10:21.239655 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:21.239924 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:21.241101 10080 hybrid_clock.cc:648] HybridClock initialized: now 1790208621241084 us; error 27 us; skew 500 ppm
I20260924 00:10:21.242385 10080 webserver.cc:460] Webserver started at http://127.0.98.62:33095/ using document root <none> and password file <none>
I20260924 00:10:21.242579 10080 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:21.242621 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:21.242710 10080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:21.243839 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "f4cfdc28eaec446aadcfd1669907c2d2"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "73361ca779ede4bfb706b7dec329988f"
server_key_iv: "d259e6c8a1d84b0c4660635d70d6fbfb"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.244143 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4cfdc28eaec446aadcfd1669907c2d2"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "73361ca779ede4bfb706b7dec329988f"
server_key_iv: "d259e6c8a1d84b0c4660635d70d6fbfb"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.245407 10080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:10:21.246071 10094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.246275 10080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:21.246366 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "f4cfdc28eaec446aadcfd1669907c2d2"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "73361ca779ede4bfb706b7dec329988f"
server_key_iv: "d259e6c8a1d84b0c4660635d70d6fbfb"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.246423 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:21.249663 10080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:21.249886 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:21.249980 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:21.253644 10080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44075
I20260924 00:10:21.253700 10146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44075 every 8 connection(s)
I20260924 00:10:21.254063 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:21.254740 10147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:21.257656 10147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2: Bootstrap starting.
I20260924 00:10:21.258255 10147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:21.258569 10147 log.cc:824] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:21.259325 10147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2: No bootstrap required, opened a new log
I20260924 00:10:21.260888 10147 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } }
I20260924 00:10:21.261030 10147 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:21.261078 10147 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cfdc28eaec446aadcfd1669907c2d2, State: Initialized, Role: FOLLOWER
I20260924 00:10:21.261235 10147 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } }
I20260924 00:10:21.261296 10147 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:21.261346 10147 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:21.261396 10147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:21.261922 10147 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } }
I20260924 00:10:21.262037 10147 leader_election.cc:302] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4cfdc28eaec446aadcfd1669907c2d2; no voters:
I20260924 00:10:21.262238 10147 leader_election.cc:288] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:10:21.262326 10150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:21.262490 10150 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [term 1 LEADER]: Becoming Leader. State: Replica: f4cfdc28eaec446aadcfd1669907c2d2, State: Running, Role: LEADER
I20260924 00:10:21.262504 10147 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:21.262581 10150 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } }
I20260924 00:10:21.262957 10152 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4cfdc28eaec446aadcfd1669907c2d2. Latest consensus state: current_term: 1 leader_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } } }
I20260924 00:10:21.263041 10152 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:21.263195 10151 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cfdc28eaec446aadcfd1669907c2d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44075 } } }
I20260924 00:10:21.263278 10151 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:21.263573 10158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:21.264001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10080
I20260924 00:10:21.264101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:10:21.264125 10158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:21.264320 392 external_mini_cluster.cc:1428] Setting key 591c368d53c7ce959d2c9df4e903b2a5
I20260924 00:10:21.265656 10158 catalog_manager.cc:1347] Generated new cluster ID: ddc27d08af0d4229af827d16f91a9e27
I20260924 00:10:21.265712 10158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:10:21.269079 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.264808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.98.62:44075)
0924 00:10:21.267583 (+ 2775us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.267591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.267616 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:10:21.267712 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.267717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.267752 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.267887 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.268632 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:21.268635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:21.268635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:21.268638 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:21.268664 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:21.268675 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:21.268747 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:21.268842 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:10:21.268904 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2722,"thread_start_us":65,"threads_started":1}
I20260924 00:10:21.278585 10158 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:10:21.278777 10158 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:21.284024 10158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4cfdc28eaec446aadcfd1669907c2d2: Generated new TSK 0
I20260924 00:10:21.284247 10158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:10:21.286401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:10:21.375885 10171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:21.376135 10171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:21.376178 10171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:21.377435 10171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:21.377502 10171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:21.377547 10171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:21.378937 10171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:21.379210 10171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:21.379572 10171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:21.384506 10177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.384545 10179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.384563 10176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.384850 10171 server_base.cc:1034] running on GCE node
I20260924 00:10:21.385079 10171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:21.385331 10171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:21.386498 10171 hybrid_clock.cc:648] HybridClock initialized: now 1790208621386484 us; error 31 us; skew 500 ppm
I20260924 00:10:21.392762 10171 webserver.cc:460] Webserver started at http://127.0.98.1:40645/ using document root <none> and password file <none>
I20260924 00:10:21.393019 10171 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:21.393091 10171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:21.393251 10171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:21.394354 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "13070ef1ec2246a2b0337e33a97d1bc1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "d70e4eea098098fe5333b5659215ff33"
server_key_iv: "f10b5a1ad6b3d451ac745115579f8a32"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.394732 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13070ef1ec2246a2b0337e33a97d1bc1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "d70e4eea098098fe5333b5659215ff33"
server_key_iv: "f10b5a1ad6b3d451ac745115579f8a32"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.396301 10171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:10:21.397152 10185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.397372 10171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:10:21.397467 10171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "13070ef1ec2246a2b0337e33a97d1bc1"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "d70e4eea098098fe5333b5659215ff33"
server_key_iv: "f10b5a1ad6b3d451ac745115579f8a32"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.397535 10171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:21.402040 10171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:21.402315 10171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:21.402439 10171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:21.402629 10171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:21.402992 10171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:21.403036 10171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.403064 10171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:21.403084 10171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.409706 10171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43287
I20260924 00:10:21.409796 10298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43287 every 8 connection(s)
I20260924 00:10:21.410140 10171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:10:21.411679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10171
I20260924 00:10:21.411755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:10:21.411950 392 external_mini_cluster.cc:1428] Setting key fd2464c023aab2d479199f4fb83fd519
I20260924 00:10:21.412955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 00:10:21.413563 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.410907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49483 (local address 127.0.98.62:44075)
0924 00:10:21.410976 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.410978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.411251 (+ 273us) server_negotiation.cc:407] Connection header received
0924 00:10:21.411826 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.411827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.411835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.411896 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.413290 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:21.413292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:21.413292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:21.413296 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:21.413322 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:21.413327 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:21.413398 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:21.413479 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 00:10:21.413494 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:10:21.413776 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.410882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44075 (local address 127.0.98.1:49483)
0924 00:10:21.411095 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:21.411137 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:21.411676 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:21.412007 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:21.412012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:21.412022 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:21.413124 (+ 1102us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:21.413125 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:21.413170 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:21.413200 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:21.413397 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:21.413404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:21.413407 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:10:21.413490 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:10:21.413551 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260924 00:10:21.414203 10299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44075
I20260924 00:10:21.414315 10299 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:21.414466 10299 heartbeater.cc:511] Master 127.0.98.62:44075 requested a full tablet report, sending...
I20260924 00:10:21.414947 10111 ts_manager.cc:194] Registered new tserver with Master: 13070ef1ec2246a2b0337e33a97d1bc1 (127.0.98.1:43287)
W20260924 00:10:21.494596 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:21.494830 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:21.494854 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:21.496234 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:21.496295 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:21.496331 10302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:21.497836 10302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:21.498118 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:21.498502 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:21.504004 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.503957 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.503957 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.504343 10302 server_base.cc:1034] running on GCE node
I20260924 00:10:21.504551 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:21.504765 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:21.505944 10302 hybrid_clock.cc:648] HybridClock initialized: now 1790208621505935 us; error 45 us; skew 500 ppm
I20260924 00:10:21.507272 10302 webserver.cc:460] Webserver started at http://127.0.98.2:46677/ using document root <none> and password file <none>
I20260924 00:10:21.507537 10302 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:21.507609 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:21.507766 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:21.508873 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "b79915fb877c4b3382f48be29428a0a3"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "82d55066a9e7d6977288ee088a9e7aee"
server_key_iv: "278c89adfa0744078336a132ae77b165"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.509251 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b79915fb877c4b3382f48be29428a0a3"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "82d55066a9e7d6977288ee088a9e7aee"
server_key_iv: "278c89adfa0744078336a132ae77b165"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.510609 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:10:21.511430 10316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.511602 10302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:21.511719 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "b79915fb877c4b3382f48be29428a0a3"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "82d55066a9e7d6977288ee088a9e7aee"
server_key_iv: "278c89adfa0744078336a132ae77b165"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.511801 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:21.515291 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:21.515600 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:21.515753 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:21.515977 10302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:21.516294 10302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:21.516350 10302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.516390 10302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:21.516419 10302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.522346 10302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260924 00:10:21.522384 10429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260924 00:10:21.522727 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:10:21.525934 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.523508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57595 (local address 127.0.98.62:44075)
0924 00:10:21.523555 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.523557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.523813 (+ 256us) server_negotiation.cc:407] Connection header received
0924 00:10:21.524346 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.524348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.524355 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.524419 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.525671 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:21.525672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:21.525673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:21.525678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:21.525696 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:21.525701 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:21.525755 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:21.525845 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 00:10:21.525861 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:10:21.526160 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.523495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44075 (local address 127.0.98.2:57595)
0924 00:10:21.523691 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:21.523724 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:21.524216 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:21.524500 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:21.524504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:21.524512 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:21.525509 (+ 997us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:21.525509 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:21.525540 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:21.525569 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:21.525751 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:21.525755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:21.525759 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:21.525857 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 00:10:21.525922 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260924 00:10:21.526684 10430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44075
I20260924 00:10:21.526762 10430 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:21.526925 10430 heartbeater.cc:511] Master 127.0.98.62:44075 requested a full tablet report, sending...
I20260924 00:10:21.527334 10111 ts_manager.cc:194] Registered new tserver with Master: b79915fb877c4b3382f48be29428a0a3 (127.0.98.2:42733)
I20260924 00:10:21.528223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10302
I20260924 00:10:21.528333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:10:21.528524 392 external_mini_cluster.cc:1428] Setting key a8ff7a4c83cdfcbd58a2c422a0b450c4
I20260924 00:10:21.529326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--builtin_ntp_servers=127.0.98.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:10:21.611093 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:21.611327 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:21.611351 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:21.612458 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:21.612493 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:21.612519 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:21.613778 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:21.613965 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:21.614214 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:21.619352 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.619423 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:21.619362 10441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:21.619536 10433 server_base.cc:1034] running on GCE node
I20260924 00:10:21.619882 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:21.620095 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:21.621246 10433 hybrid_clock.cc:648] HybridClock initialized: now 1790208621621234 us; error 31 us; skew 500 ppm
I20260924 00:10:21.622474 10433 webserver.cc:460] Webserver started at http://127.0.98.3:46625/ using document root <none> and password file <none>
I20260924 00:10:21.622730 10433 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:21.622798 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:21.622970 10433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:21.624140 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b72e16d7e094e03832928db0a3c9b27"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "5f0ac0b4b7997d7043b7c42bcd84fb1c"
server_key_iv: "66de13a36c2f88f9fcf8f2224fc305b3"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.624528 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b72e16d7e094e03832928db0a3c9b27"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "5f0ac0b4b7997d7043b7c42bcd84fb1c"
server_key_iv: "66de13a36c2f88f9fcf8f2224fc305b3"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.625844 10433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:10:21.626561 10447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.626731 10433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:10:21.626847 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "9b72e16d7e094e03832928db0a3c9b27"
format_stamp: "Formatted at 2026-09-24 00:10:21 on dist-test-slave-3q8k"
server_key: "5f0ac0b4b7997d7043b7c42bcd84fb1c"
server_key_iv: "66de13a36c2f88f9fcf8f2224fc305b3"
server_key_version: "encryptionkey@0"
I20260924 00:10:21.626920 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:21.630273 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:21.630535 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:21.630684 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:21.630893 10433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:21.631192 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:21.631264 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.631304 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:21.631332 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:21.636968 10433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41755
I20260924 00:10:21.637002 10560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41755 every 8 connection(s)
I20260924 00:10:21.637332 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:10:21.640152 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.638046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55891 (local address 127.0.98.62:44075)
0924 00:10:21.638113 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.638116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.638334 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:10:21.638775 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.638776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.638781 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.638846 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.639966 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:21.639967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:21.639967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:10:21.639971 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:21.639984 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:10:21.639988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:21.640040 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:21.640088 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 00:10:21.640100 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:10:21.640363 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:21.638041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44075 (local address 127.0.98.3:55891)
0924 00:10:21.638213 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:21.638247 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:21.638693 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:21.638897 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:21.638900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:21.638909 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:21.639823 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:10:21.639823 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:21.639852 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:10:21.639880 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:21.640041 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:21.640043 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:21.640045 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:10:21.640094 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:10:21.640140 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260924 00:10:21.640798 10561 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44075
I20260924 00:10:21.640875 10561 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:21.641040 10561 heartbeater.cc:511] Master 127.0.98.62:44075 requested a full tablet report, sending...
I20260924 00:10:21.641438 10111 ts_manager.cc:194] Registered new tserver with Master: 9b72e16d7e094e03832928db0a3c9b27 (127.0.98.3:41755)
I20260924 00:10:21.643896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10433
I20260924 00:10:21.643992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:10:21.644192 392 external_mini_cluster.cc:1428] Setting key 7520ea9e9db3575a699dee01e7aed136
I20260924 00:10:21.645087 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:10:21.646881 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:10:21.646393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.98.62:44075)
0924 00:10:21.646445 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:21.646447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:21.646564 (+ 117us) server_negotiation.cc:407] Connection header received
0924 00:10:21.646612 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:21.646612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:21.646616 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:21.646668 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:21.646756 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46170: BlockingRecv error: recv got EOF from 127.0.0.1:46170 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:10:21.646945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10171
I20260924 00:10:21.651743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10302
I20260924 00:10:21.655884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10433
I20260924 00:10:21.660594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10080
2026-09-24T00:10:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (536 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:10:21 dist-test-slave-3q8k krb5kdc[10571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:10:21 dist-test-slave-3q8k krb5kdc[10571](info): set up 2 sockets
Sep 24 00:10:21 dist-test-slave-3q8k krb5kdc[10571](info): commencing operation
krb5kdc: starting...
W20260924 00:10:22.716267 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:10:22 dist-test-slave-3q8k krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:10:22.846208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:22.931023 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:22.931313 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:22.931376 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:22.932641 10584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:10:22.932689 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:22.932704 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:22.932717 10584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:10:22.932773 10584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:10:22.934316 10584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:22.934585 10584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:22.934830 10584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:22.939929 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:22.939881 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:22.939992 10584 server_base.cc:1034] running on GCE node
W20260924 00:10:22.939956 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:22.940470 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:22.940785 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:22.941949 10584 hybrid_clock.cc:648] HybridClock initialized: now 1790208622941925 us; error 39 us; skew 500 ppm
Sep 24 00:10:22 dist-test-slave-3q8k krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:22.945052 10584 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:10:22.945549 10584 webserver.cc:460] Webserver started at http://127.0.98.62:39015/ using document root <none> and password file <none>
I20260924 00:10:22.945801 10584 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:22.945848 10584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:22.945992 10584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:22.947073 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "608b75841ee1469b9d61fffc1d29506f"
format_stamp: "Formatted at 2026-09-24 00:10:22 on dist-test-slave-3q8k"
server_key: "68d3c3d58f29a85edb6c50801864e392"
server_key_iv: "4e3da4f4db2e2463b7dc22c8435de570"
server_key_version: "encryptionkey@0"
I20260924 00:10:22.947486 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "608b75841ee1469b9d61fffc1d29506f"
format_stamp: "Formatted at 2026-09-24 00:10:22 on dist-test-slave-3q8k"
server_key: "68d3c3d58f29a85edb6c50801864e392"
server_key_iv: "4e3da4f4db2e2463b7dc22c8435de570"
server_key_version: "encryptionkey@0"
I20260924 00:10:22.948966 10584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:22.949733 10599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:22.949949 10584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:22.950017 10584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "608b75841ee1469b9d61fffc1d29506f"
format_stamp: "Formatted at 2026-09-24 00:10:22 on dist-test-slave-3q8k"
server_key: "68d3c3d58f29a85edb6c50801864e392"
server_key_iv: "4e3da4f4db2e2463b7dc22c8435de570"
server_key_version: "encryptionkey@0"
I20260924 00:10:22.950110 10584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:22.971375 10584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:22.973470 10584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:22.973634 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:22.977802 10584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35177
I20260924 00:10:22.977885 10651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35177 every 8 connection(s)
I20260924 00:10:22.978283 10584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:10:22.979501 10652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:22.980988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10584
I20260924 00:10:22.981060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:10:22.981266 392 external_mini_cluster.cc:1428] Setting key 42f9e9ffa5038274f1467aaa324ec9b8
I20260924 00:10:22.982792 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: Bootstrap starting.
I20260924 00:10:22.983592 10652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:22.983954 10652 log.cc:824] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:22.984781 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: No bootstrap required, opened a new log
I20260924 00:10:22.987406 10652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 00:10:22.987601 10652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:22.987699 10652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608b75841ee1469b9d61fffc1d29506f, State: Initialized, Role: FOLLOWER
I20260924 00:10:22.987887 10652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 00:10:22.987991 10652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:10:22.988044 10652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:10:22.988101 10652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:22.988891 10652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 00:10:22.989024 10652 leader_election.cc:302] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608b75841ee1469b9d61fffc1d29506f; no voters:
I20260924 00:10:22.989231 10652 leader_election.cc:288] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:10:22.989259 10657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 00:10:22 dist-test-slave-3q8k krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:22.989499 10657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [term 1 LEADER]: Becoming Leader. State: Replica: 608b75841ee1469b9d61fffc1d29506f, State: Running, Role: LEADER
I20260924 00:10:22.989532 10652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:10:22.989598 10657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } }
I20260924 00:10:22.989916 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608b75841ee1469b9d61fffc1d29506f. Latest consensus state: current_term: 1 leader_uuid: "608b75841ee1469b9d61fffc1d29506f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } } }
I20260924 00:10:22.990011 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:22.989928 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608b75841ee1469b9d61fffc1d29506f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608b75841ee1469b9d61fffc1d29506f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35177 } } }
I20260924 00:10:22.990161 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f [sys.catalog]: This master's current role is: LEADER
I20260924 00:10:22.990324 10662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:10:22.991308 10662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:10:22.994369 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:22.981838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.0.98.62:35177)
0924 00:10:22.982062 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:22.982074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:22.982110 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:10:22.982715 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:22.982725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:22.982795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:22.982964 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:22.983657 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:22.985222 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:22.985982 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:22.986737 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:22.986910 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:22.989986 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:22.989990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:22.989991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:22.990005 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:22.992556 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:22.992971 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:22.992975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:22.992976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:22.993107 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:22.993307 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:22.993308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:22.993308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:22.993668 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:22.993822 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:22.993935 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:10:22.994089 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
I20260924 00:10:22.995672 10662 catalog_manager.cc:1347] Generated new cluster ID: 0f97a51cdae943c285d12ac448063808
I20260924 00:10:22.995736 10662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:10:23.005959 10662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:10:23.006768 10662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:10:23.013108 10662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: Generated new TSK 0
I20260924 00:10:23.013340 10662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:10:23.055193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:23.142056 10678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:23.142309 10678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:23.142351 10678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:23.143620 10678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:23.143685 10678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:23.143730 10678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:10:23.145257 10678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:23.145526 10678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:23.145763 10678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:23.150847 10683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.150893 10684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.150929 10686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:23.151122 10678 server_base.cc:1034] running on GCE node
I20260924 00:10:23.151358 10678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:23.151614 10678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:23.152772 10678 hybrid_clock.cc:648] HybridClock initialized: now 1790208623152759 us; error 30 us; skew 500 ppm
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:23.155532 10678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:10:23.156080 10678 webserver.cc:460] Webserver started at http://127.0.98.1:37747/ using document root <none> and password file <none>
I20260924 00:10:23.156339 10678 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:23.156409 10678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:23.156564 10678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:23.157660 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "32791b1dfec04c7cae26be95e31f4187"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "c92dda966081dbeb7d91d8ed58dce860"
server_key_iv: "0b8642bbecff56b0574ab8a58548f6ba"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.158031 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32791b1dfec04c7cae26be95e31f4187"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "c92dda966081dbeb7d91d8ed58dce860"
server_key_iv: "0b8642bbecff56b0574ab8a58548f6ba"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.159483 10678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:10:23.160260 10693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.160418 10678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:23.160506 10678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "32791b1dfec04c7cae26be95e31f4187"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "c92dda966081dbeb7d91d8ed58dce860"
server_key_iv: "0b8642bbecff56b0574ab8a58548f6ba"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.160580 10678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:23.182907 10678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:23.185046 10678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:23.185235 10678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:23.185499 10678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:23.185820 10678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:23.185881 10678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.185923 10678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:23.185953 10678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.192307 10678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42035
I20260924 00:10:23.192356 10806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42035 every 8 connection(s)
I20260924 00:10:23.192678 10678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:23.201315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10678
I20260924 00:10:23.201395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:10:23.201648 392 external_mini_cluster.cc:1428] Setting key e307f0bc4aabf1c157bbf2c772f6c24a
I20260924 00:10:23.202816 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.193384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53179 (local address 127.0.98.62:35177)
0924 00:10:23.193436 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.193438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.193632 (+ 194us) server_negotiation.cc:407] Connection header received
0924 00:10:23.194136 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.194137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.194146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.194206 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:23.195168 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.195779 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.196769 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.196973 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.197038 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.199189 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:23.199191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:23.199191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:23.199198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:23.201081 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.201570 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.201572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.201572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.201641 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.201969 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.201970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.201970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.202120 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:23.202196 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.202570 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.202677 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:10:23.202818 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.193350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35177 (local address 127.0.98.1:53179)
0924 00:10:23.193510 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.193552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.194049 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.194285 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.194289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.194881 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:23.195072 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.195077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.195873 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.195873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.196665 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.196667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.197054 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.197055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.197153 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.197645 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:23.197653 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:23.199059 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:23.201209 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.201214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.201216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.201496 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.201732 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.201733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.201733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.201889 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.202214 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:23.202220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:23.202404 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.202526 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.202651 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260924 00:10:23.204236 10807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35177
I20260924 00:10:23.204370 10807 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:23.204561 10807 heartbeater.cc:511] Master 127.0.98.62:35177 requested a full tablet report, sending...
I20260924 00:10:23.205143 10616 ts_manager.cc:194] Registered new tserver with Master: 32791b1dfec04c7cae26be95e31f4187 (127.0.98.1:42035)
I20260924 00:10:23.205910 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53179
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:10:23.226214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:23.307850 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:23.308058 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:23.308082 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:23.309231 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:23.309275 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:23.309301 10812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:10:23.310779 10812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:23.310972 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:23.311173 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:23.316177 10818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.316177 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.316259 10820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:23.316758 10812 server_base.cc:1034] running on GCE node
I20260924 00:10:23.316960 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:23.317163 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:23.318315 10812 hybrid_clock.cc:648] HybridClock initialized: now 1790208623318297 us; error 42 us; skew 500 ppm
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:23.321107 10812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:10:23.321522 10812 webserver.cc:460] Webserver started at http://127.0.98.2:44929/ using document root <none> and password file <none>
I20260924 00:10:23.321707 10812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:23.321748 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:23.321838 10812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:23.322880 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "a93133e62dfa421c8b3216175d3952c7"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "9006f6f83fc2967ece7653fe203622e5"
server_key_iv: "5cbf5daa4bb98d2689c7bf45c827ac0a"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.323175 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a93133e62dfa421c8b3216175d3952c7"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "9006f6f83fc2967ece7653fe203622e5"
server_key_iv: "5cbf5daa4bb98d2689c7bf45c827ac0a"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.324517 10812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:23.325176 10827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.325376 10812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:10:23.325461 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "a93133e62dfa421c8b3216175d3952c7"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "9006f6f83fc2967ece7653fe203622e5"
server_key_iv: "5cbf5daa4bb98d2689c7bf45c827ac0a"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.325517 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:23.345616 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:23.347992 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:23.348111 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:23.348299 10812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:23.348582 10812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:23.348618 10812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.348641 10812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:23.348654 10812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.354797 10812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36331
I20260924 00:10:23.354866 10940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36331 every 8 connection(s)
I20260924 00:10:23.355196 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:10:23.361171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10812
I20260924 00:10:23.361292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:10:23.361496 392 external_mini_cluster.cc:1428] Setting key ba2cdcd215e8bc54e45c79d40a1c08cf
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:23.365181 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.356030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55633 (local address 127.0.98.62:35177)
0924 00:10:23.356087 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.356088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.356332 (+ 244us) server_negotiation.cc:407] Connection header received
0924 00:10:23.356872 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.356873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.356885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.356937 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:23.357906 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.358410 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.359476 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.359679 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.359788 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.362124 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:23.362125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:23.362126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:23.362134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:23.363717 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.364153 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.364155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.364155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.364217 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.364519 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.364520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.364521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.364642 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:23.364721 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.364998 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.365083 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:10:23.365464 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.356018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35177 (local address 127.0.98.2:55633)
0924 00:10:23.356202 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.356237 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.356755 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.357021 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.357024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.357586 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:23.357788 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.357793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.358521 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.358522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.359392 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.359394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.359794 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.359795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.359935 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.360473 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:23.360481 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:23.362009 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:23.363815 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.363823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.363825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.364057 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.364287 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.364287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.364288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.364442 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.364711 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:23.364718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:23.364867 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.364978 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.365164 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260924 00:10:23.365965 10941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35177
I20260924 00:10:23.366075 10941 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:23.366257 10941 heartbeater.cc:511] Master 127.0.98.62:35177 requested a full tablet report, sending...
I20260924 00:10:23.366716 10616 ts_manager.cc:194] Registered new tserver with Master: a93133e62dfa421c8b3216175d3952c7 (127.0.98.2:36331)
I20260924 00:10:23.367324 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55633
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:10:23.386399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:10:23.466521 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:10:23.466728 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:10:23.466751 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:10:23.467995 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:10:23.468043 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:10:23.468070 10946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:10:23.469482 10946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:10:23.469668 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:10:23.469857 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:10:23.474751 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.474753 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:10:23.474751 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:10:23.474884 10946 server_base.cc:1034] running on GCE node
I20260924 00:10:23.475436 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:10:23.475745 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:10:23.476912 10946 hybrid_clock.cc:648] HybridClock initialized: now 1790208623476907 us; error 36 us; skew 500 ppm
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:10:23.479488 10946 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:10:23.479938 10946 webserver.cc:460] Webserver started at http://127.0.98.3:40651/ using document root <none> and password file <none>
I20260924 00:10:23.480162 10946 fs_manager.cc:359] Metadata directory not provided
I20260924 00:10:23.480229 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:10:23.480381 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:10:23.481415 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "003abe49c24944fa9cd2185e4b386e48"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "ec0f3cb666d97d7f6bb1a001ea429d3e"
server_key_iv: "c5cfa9e6b6735a2d0a872a831a4df1d2"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.481799 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "003abe49c24944fa9cd2185e4b386e48"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "ec0f3cb666d97d7f6bb1a001ea429d3e"
server_key_iv: "c5cfa9e6b6735a2d0a872a831a4df1d2"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.483176 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:10:23.484015 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.484182 10946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:10:23.484279 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "003abe49c24944fa9cd2185e4b386e48"
format_stamp: "Formatted at 2026-09-24 00:10:23 on dist-test-slave-3q8k"
server_key: "ec0f3cb666d97d7f6bb1a001ea429d3e"
server_key_iv: "c5cfa9e6b6735a2d0a872a831a4df1d2"
server_key_version: "encryptionkey@0"
I20260924 00:10:23.484364 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:10:23.501324 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:10:23.503790 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:10:23.503971 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:10:23.504216 10946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:10:23.504542 10946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:10:23.504595 10946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.504637 10946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:10:23.504664 10946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:10:23.510648 10946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44127
I20260924 00:10:23.510694 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44127 every 8 connection(s)
I20260924 00:10:23.511018 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:10:23.519917 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.511768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38291 (local address 127.0.98.62:35177)
0924 00:10:23.511838 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.511840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.512015 (+ 175us) server_negotiation.cc:407] Connection header received
0924 00:10:23.512479 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.512480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.512489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.512566 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:23.513683 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.514174 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.515151 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.515395 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.515450 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.517497 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:23.517498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:23.517498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:23.517505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:23.518650 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.519032 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.519033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.519034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.519093 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.519371 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.519371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.519372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.519487 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:23.519543 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.519779 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.519852 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:10:23.520074 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.511721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35177 (local address 127.0.98.3:38291)
0924 00:10:23.511883 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.511926 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.512399 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.512655 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.512660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.513391 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:10:23.513588 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.513593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.514264 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.514265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.515074 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.515076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.515466 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.515467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.515565 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.516029 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:10:23.516037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:10:23.517408 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:10:23.518721 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.518726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.518728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.518944 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.519162 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:10:23.519163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:10:23.519163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:10:23.519322 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:10:23.519547 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:10:23.519551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:10:23.519661 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.519743 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.519856 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 00:10:23.520583 11075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35177
I20260924 00:10:23.520689 11075 heartbeater.cc:464] Registering TS with master...
I20260924 00:10:23.520864 11075 heartbeater.cc:511] Master 127.0.98.62:35177 requested a full tablet report, sending...
I20260924 00:10:23.521030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 10946
I20260924 00:10:23.521142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:10:23.521263 10616 ts_manager.cc:194] Registered new tserver with Master: 003abe49c24944fa9cd2185e4b386e48 (127.0.98.3:44127)
I20260924 00:10:23.521338 392 external_mini_cluster.cc:1428] Setting key c625169c4cf35755419b8a2bc068b714
I20260924 00:10:23.521761 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38291
I20260924 00:10:23.522221 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:10:23.529349 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.523505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.0.98.62:35177)
0924 00:10:23.523573 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.523575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.523677 (+ 102us) server_negotiation.cc:407] Connection header received
0924 00:10:23.523705 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.523705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.523714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.523770 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:10:23.524265 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.524738 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.525454 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.525652 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.525720 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.526919 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:10:23.526920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:10:23.526920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:10:23.526926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:10:23.528343 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.528635 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.528636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.528636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.528685 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:10:23.528813 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:10:23.528813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:10:23.528814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:10:23.528945 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:10:23.529018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.529135 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.529203 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:10:23.531013 10616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:10:23.532627 10616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:10:23.539266 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.535029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36331 (local address 127.0.0.1:47994)
0924 00:10:23.535625 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.535637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.535690 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.535967 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.535968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.535970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:23.536012 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.536013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.536956 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.536957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.538062 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.538064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.538927 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.539010 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.539023 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.539098 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.539159 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":59,"thread_start_us":66,"threads_started":1}
I20260924 00:10:23.539292 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.535135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.0.98.2:36331)
0924 00:10:23.535299 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.535303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.535748 (+ 445us) server_negotiation.cc:407] Connection header received
0924 00:10:23.535792 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.535794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.535810 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.535882 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:23.536094 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.536861 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538155 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.538833 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538909 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.538966 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.539126 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.539167 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260924 00:10:23.539446 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.534991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.1:42035)
0924 00:10:23.535377 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.535382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.535392 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:10:23.535428 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.535430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.535444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.535520 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:23.535786 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.536880 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538392 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.538899 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538969 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.539027 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.539309 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.539353 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":88,"threads_started":1}
I20260924 00:10:23.539459 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.534879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42035 (local address 127.0.0.1:33428)
0924 00:10:23.535166 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.535199 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.535287 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.535654 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.535654 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.535658 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:23.535701 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.535703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.537002 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.537003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.538287 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.538290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.539059 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.539060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.539119 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.539122 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.539166 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.539194 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":49,"thread_start_us":77,"threads_started":1}
I20260924 00:10:23.540082 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.534854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.0.1:53930)
0924 00:10:23.535155 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.535198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.535289 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.535534 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.535537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.535539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:23.535619 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.535621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.537002 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.537003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.538287 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.538290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.539245 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.539246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.539311 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.539914 (+ 603us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.539973 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.540016 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":85,"thread_start_us":41,"threads_started":1}
I20260924 00:10:23.540112 10875 tablet_service.cc:1467] Processing CreateTablet for tablet 1064eb80de97457da111c9d813b8297c (DEFAULT_TABLE table=test-table [id=b1540685901d4a3aa635847c841556d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:23.540202 10741 tablet_service.cc:1467] Processing CreateTablet for tablet 1064eb80de97457da111c9d813b8297c (DEFAULT_TABLE table=test-table [id=b1540685901d4a3aa635847c841556d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:23.540362 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.534923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.98.3:44127)
0924 00:10:23.535169 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.535174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.535375 (+ 201us) server_negotiation.cc:407] Connection header received
0924 00:10:23.535421 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.535423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.535439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.535520 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:23.535710 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.536880 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.538377 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.539059 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.539144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.539193 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.540231 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.540273 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260924 00:10:23.540504 10875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1064eb80de97457da111c9d813b8297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:23.540547 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1064eb80de97457da111c9d813b8297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:23.540925 11009 tablet_service.cc:1467] Processing CreateTablet for tablet 1064eb80de97457da111c9d813b8297c (DEFAULT_TABLE table=test-table [id=b1540685901d4a3aa635847c841556d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:10:23.541224 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1064eb80de97457da111c9d813b8297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:10:23.543107 11092 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Bootstrap starting.
I20260924 00:10:23.543797 11092 tablet_bootstrap.cc:654] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:23.543856 11093 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Bootstrap starting.
I20260924 00:10:23.544346 11094 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Bootstrap starting.
I20260924 00:10:23.544528 11093 tablet_bootstrap.cc:654] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:23.544567 11092 log.cc:824] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:23.545017 11094 tablet_bootstrap.cc:654] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Neither blocks nor log segments found. Creating new log.
I20260924 00:10:23.545094 11093 log.cc:824] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:23.545377 11092 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: No bootstrap required, opened a new log
I20260924 00:10:23.545470 11092 ts_tablet_manager.cc:1397] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:10:23.545598 11094 log.cc:824] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Log is configured to *not* fsync() on all Append() calls
I20260924 00:10:23.545929 11093 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: No bootstrap required, opened a new log
I20260924 00:10:23.546044 11093 ts_tablet_manager.cc:1397] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:10:23.546211 11094 tablet_bootstrap.cc:492] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: No bootstrap required, opened a new log
I20260924 00:10:23.546262 11094 ts_tablet_manager.cc:1397] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:10:23.547567 11092 raft_consensus.cc:348] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.547763 11092 raft_consensus.cc:374] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:23.547799 11092 raft_consensus.cc:729] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93133e62dfa421c8b3216175d3952c7, State: Initialized, Role: FOLLOWER
I20260924 00:10:23.547943 11092 consensus_queue.cc:260] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.548210 11092 ts_tablet_manager.cc:1428] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:10:23.548136 11093 raft_consensus.cc:348] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.548141 11094 raft_consensus.cc:348] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.548288 11093 raft_consensus.cc:374] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:23.548291 11094 raft_consensus.cc:374] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:10:23.548333 11094 raft_consensus.cc:729] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003abe49c24944fa9cd2185e4b386e48, State: Initialized, Role: FOLLOWER
I20260924 00:10:23.548333 11093 raft_consensus.cc:729] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32791b1dfec04c7cae26be95e31f4187, State: Initialized, Role: FOLLOWER
I20260924 00:10:23.548359 10941 heartbeater.cc:503] Master 127.0.98.62:35177 was elected leader, sending a full tablet report...
I20260924 00:10:23.548462 11093 consensus_queue.cc:260] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.548473 11094 consensus_queue.cc:260] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.548738 11075 heartbeater.cc:503] Master 127.0.98.62:35177 was elected leader, sending a full tablet report...
I20260924 00:10:23.548955 11094 ts_tablet_manager.cc:1428] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:10:23.548907 11093 ts_tablet_manager.cc:1428] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:10:23.549360 10807 heartbeater.cc:503] Master 127.0.98.62:35177 was elected leader, sending a full tablet report...
W20260924 00:10:23.605949 10942 tablet.cc:2369] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:10:23.693392 10808 tablet.cc:2369] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:10:23.753676 11098 raft_consensus.cc:482] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:10:23.753804 11098 raft_consensus.cc:504] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.754169 11098 leader_election.cc:288] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 003abe49c24944fa9cd2185e4b386e48 (127.0.98.3:44127), 32791b1dfec04c7cae26be95e31f4187 (127.0.98.1:42035)
I20260924 00:10:23.757221 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.754420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43621 (local address 127.0.98.1:42035)
0924 00:10:23.754467 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.754468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.754843 (+ 375us) server_negotiation.cc:407] Connection header received
0924 00:10:23.754891 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.754892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.754903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.754939 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:23.755083 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.755626 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.756543 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.756879 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.756940 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.756948 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.757108 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.757152 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:10:23.757233 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.754390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42035 (local address 127.0.98.2:43621)
0924 00:10:23.754752 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.754761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.754830 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.754943 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.754944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.754945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:23.755007 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.755009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.755702 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.755703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.756456 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.756458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.756956 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.756957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.757032 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.757036 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.757093 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.757122 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":171,"thread_start_us":114,"threads_started":1}
I20260924 00:10:23.757298 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.754426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46535 (local address 127.0.98.3:44127)
0924 00:10:23.754473 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.754475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.754489 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:10:23.754586 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.754587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.754596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.754648 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:10:23.754854 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.755498 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.756447 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.756937 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.757006 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.757015 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.757184 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.757219 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:10:23.757548 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.754390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.98.2:46535)
0924 00:10:23.754445 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.754458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:10:23.754507 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:10:23.754678 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:10:23.754679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:10:23.754681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:10:23.754761 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.754762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.755571 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.755572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.756360 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:10:23.756362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.757019 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.757020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:10:23.757107 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.757111 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:10:23.757166 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 00:10:23.757203 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 00:10:23.757659 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1064eb80de97457da111c9d813b8297c" candidate_uuid: "a93133e62dfa421c8b3216175d3952c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32791b1dfec04c7cae26be95e31f4187" is_pre_election: true
I20260924 00:10:23.757820 10761 raft_consensus.cc:2454] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93133e62dfa421c8b3216175d3952c7 in term 0.
I20260924 00:10:23.757877 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1064eb80de97457da111c9d813b8297c" candidate_uuid: "a93133e62dfa421c8b3216175d3952c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003abe49c24944fa9cd2185e4b386e48" is_pre_election: true
I20260924 00:10:23.758023 11029 raft_consensus.cc:2454] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a93133e62dfa421c8b3216175d3952c7 in term 0.
I20260924 00:10:23.758086 10828 leader_election.cc:302] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32791b1dfec04c7cae26be95e31f4187, a93133e62dfa421c8b3216175d3952c7; no voters:
I20260924 00:10:23.758211 11098 raft_consensus.cc:2781] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:10:23.758279 11098 raft_consensus.cc:482] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:10:23.758306 11098 raft_consensus.cc:3037] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:23.759150 11098 raft_consensus.cc:504] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
I20260924 00:10:23.759325 11098 leader_election.cc:288] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [CANDIDATE]: Term 1 election: Requested vote from peers 003abe49c24944fa9cd2185e4b386e48 (127.0.98.3:44127), 32791b1dfec04c7cae26be95e31f4187 (127.0.98.1:42035)
I20260924 00:10:23.759431 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1064eb80de97457da111c9d813b8297c" candidate_uuid: "a93133e62dfa421c8b3216175d3952c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003abe49c24944fa9cd2185e4b386e48"
I20260924 00:10:23.759472 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1064eb80de97457da111c9d813b8297c" candidate_uuid: "a93133e62dfa421c8b3216175d3952c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32791b1dfec04c7cae26be95e31f4187"
I20260924 00:10:23.759526 11029 raft_consensus.cc:3037] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:23.759536 10761 raft_consensus.cc:3037] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:10:23.760421 11029 raft_consensus.cc:2454] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93133e62dfa421c8b3216175d3952c7 in term 1.
I20260924 00:10:23.760421 10761 raft_consensus.cc:2454] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93133e62dfa421c8b3216175d3952c7 in term 1.
I20260924 00:10:23.760583 10830 leader_election.cc:302] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003abe49c24944fa9cd2185e4b386e48, a93133e62dfa421c8b3216175d3952c7; no voters:
I20260924 00:10:23.760694 11098 raft_consensus.cc:2781] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:10:23.760895 11098 raft_consensus.cc:686] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 1 LEADER]: Becoming Leader. State: Replica: a93133e62dfa421c8b3216175d3952c7, State: Running, Role: LEADER
I20260924 00:10:23.761000 11098 consensus_queue.cc:237] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } }
W20260924 00:10:23.761556 11076 tablet.cc:2369] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:10:23.761876 10616 catalog_manager.cc:5495] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a93133e62dfa421c8b3216175d3952c7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a93133e62dfa421c8b3216175d3952c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a93133e62dfa421c8b3216175d3952c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 } health_report { overall_health: UNKNOWN } } }
I20260924 00:10:23.768793 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.766544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.0.98.2:36331)
0924 00:10:23.766609 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:10:23.766611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:10:23.766618 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:10:23.766645 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:10:23.766645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:10:23.766660 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:10:23.766717 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:10:23.766850 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.767373 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.768109 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:10:23.768283 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:10:23.768335 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:10:23.768484 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:10:23.768577 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:10:23.768634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:10:23.768662 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 00:10:23.768678 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:10:23.770514 11029 raft_consensus.cc:1261] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 1 FOLLOWER]: Refusing update from remote peer a93133e62dfa421c8b3216175d3952c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:23.770529 10761 raft_consensus.cc:1261] T 1064eb80de97457da111c9d813b8297c P 32791b1dfec04c7cae26be95e31f4187 [term 1 FOLLOWER]: Refusing update from remote peer a93133e62dfa421c8b3216175d3952c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:10:23.770844 11098 consensus_queue.cc:1035] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003abe49c24944fa9cd2185e4b386e48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:23.771008 11102 consensus_queue.cc:1035] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32791b1dfec04c7cae26be95e31f4187" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:10:23.773213 11105 mvcc.cc:205] Tried to move back new op lower bound from 7332694522961240064 to 7332694522925625344. Current Snapshot: MvccSnapshot[applied={T|T < 7332694522961240064}]
I20260924 00:10:23.773788 11106 mvcc.cc:205] Tried to move back new op lower bound from 7332694522961240064 to 7332694522925625344. Current Snapshot: MvccSnapshot[applied={T|T < 7332694522961240064}]
I20260924 00:10:23.773904 11107 mvcc.cc:205] Tried to move back new op lower bound from 7332694522961240064 to 7332694522925625344. Current Snapshot: MvccSnapshot[applied={T|T < 7332694522961240064 or (T in {7332694522961240064})}]
I20260924 00:10:23.775799 10615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:23.775898 10615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:10:23.776672 10615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 608b75841ee1469b9d61fffc1d29506f: Sending DeleteTablet for 3 replicas of tablet 1064eb80de97457da111c9d813b8297c
I20260924 00:10:23.777119 11009 tablet_service.cc:1514] Processing DeleteTablet for tablet 1064eb80de97457da111c9d813b8297c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 00:10:23.777148 10875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1064eb80de97457da111c9d813b8297c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:10:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47994
I20260924 00:10:23.777449 11116 tablet_replica.cc:330] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: stopping tablet replica
I20260924 00:10:23.777513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10678
I20260924 00:10:23.777597 11116 raft_consensus.cc:2229] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:10:23.777714 11116 raft_consensus.cc:2258] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:10:23.778721 11116 ts_tablet_manager.cc:1905] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:10:23.780611 11116 ts_tablet_manager.cc:1918] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:23.780663 11116 log.cc:1196] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/1064eb80de97457da111c9d813b8297c
I20260924 00:10:23.780872 11116 ts_tablet_manager.cc:1939] T 1064eb80de97457da111c9d813b8297c P a93133e62dfa421c8b3216175d3952c7: Deleting consensus metadata
I20260924 00:10:23.783349 10602 catalog_manager.cc:4841] TS a93133e62dfa421c8b3216175d3952c7 (127.0.98.2:36331): tablet 1064eb80de97457da111c9d813b8297c (table test-table [id=b1540685901d4a3aa635847c841556d6]) successfully deleted
I20260924 00:10:23.783864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10812
I20260924 00:10:23.785652 11115 tablet_replica.cc:330] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: stopping tablet replica
I20260924 00:10:23.785777 11115 raft_consensus.cc:2229] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:10:23.785923 11115 raft_consensus.cc:2258] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:10:23.786198 10600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
I20260924 00:10:23.786782 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.786619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42035 (local address 127.0.0.1:33430)
0924 00:10:23.786678 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.786722 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 00:10:23.786870 11115 ts_tablet_manager.cc:1905] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:10:23.786916 10600 catalog_manager.cc:4584] TS 32791b1dfec04c7cae26be95e31f4187 (127.0.98.1:42035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1064eb80de97457da111c9d813b8297c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42035: connect: Connection refused (error 111)
I20260924 00:10:23.788717 11115 ts_tablet_manager.cc:1918] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:10:23.788841 11115 log.cc:1196] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/1064eb80de97457da111c9d813b8297c
I20260924 00:10:23.789191 11115 ts_tablet_manager.cc:1939] T 1064eb80de97457da111c9d813b8297c P 003abe49c24944fa9cd2185e4b386e48: Deleting consensus metadata
I20260924 00:10:23.789202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10946
I20260924 00:10:23.793838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 10584
I20260924 00:10:23.793943 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:10:23.793815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.0.1:53936)
0924 00:10:23.793878 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:10:23.793907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
2026-09-24T00:10:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2134 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[11120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[11120](info): set up 2 sockets
Sep 24 00:10:23 dist-test-slave-3q8k krb5kdc[11120](info): commencing operation
krb5kdc: starting...
W20260924 00:10:24.836292 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:10:24 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:10:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 00:10:25.000361 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskElJ0hL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/postgres -l logfile start
2026-09-24 00:10:27.497 UTC [11150] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:10:27.497 UTC [11150] LOG: listening on IPv4 address "127.0.98.20", port 54751
2026-09-24 00:10:27.500 UTC [11150] LOG: listening on Unix socket "/tmp/.s.PGSQL.54751"
2026-09-24 00:10:27.523 UTC [11153] LOG: database system was shut down at 2026-09-24 00:10:26 UTC
2026-09-24 00:10:27.527 UTC [11150] LOG: database system is ready to accept connections
I20260924 00:10:28.476224 392 mini_postgres.cc:96] Postgres bound to 54751
2026-09-24 00:10:28.482 UTC [11161] FATAL: database "slave" does not exist
127.0.98.20:54751 - accepting connections
I20260924 00:10:28.482906 392 mini_ranger.cc:161] Starting Ranger...
I20260924 00:10:28.497869 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:10:28.985126 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:10:28.985212 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin
2026-09-24 00:10:29,269 [I] DB FLAVOR :POSTGRES
2026-09-24 00:10:29,270 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:10:29,270 [I] Checking connection..
2026-09-24 00:10:29,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:10:29,537 [I] Checking connection passed.
2026-09-24 00:10:29,537 [I] --------- Verifying version history table ---------
2026-09-24 00:10:29,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:10:29,794 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:10:29,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:10:30,029 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:10:30,029 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:10:30,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:10:30.299 UTC [11225] WARNING: there is no transaction in progress
2026-09-24 00:10:30,306 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:10:30,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:10:30,571 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:10:30,571 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:10:30,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:10:30,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:10:31,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3q8k.c.gcp-upstream.internal','N') ;"
2026-09-24 00:10:31,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:10:31,560 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:10:31,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:10:31,812 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:10:31,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:10:32,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:10:32,274 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:10:32,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:10:33.226 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.242 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.259 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.274 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.291 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.309 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.364 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.371 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.386 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.396 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.408 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.421 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.431 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.437 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.444 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:33.451 UTC [11333] WARNING: there is no transaction in progress
2026-09-24 00:10:34,176 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:10:34,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:10:34,401 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:10:34,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:10:34,647 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:10:34,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:10:34,901 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:10:34,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:10:35,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:10:35,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3q8k.c.gcp-upstream.internal';"
2026-09-24 00:10:35,646 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:10:35,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:10:35,910 [I] DB_PATCHES have already been applied
I20260924 00:10:35.915866 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskElJ0hL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:10:35.915930 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 00:10:35.918892 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51767
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:10:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 00:10:37.106806 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.624s user 0.000s sys 0.009s
Sep 24, 2026 12:10:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:10:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 12:10:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:10:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:10:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:10:58.733094 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 00:10:58 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208658, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 00:10:58.821100 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 00:10:58.831132 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 00:10:59.719282 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 00:10:59.719376 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms
I20260924 00:10:59.719396 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54751
2026-09-24 00:10:59,902 [I] DB FLAVOR :POSTGRES
2026-09-24 00:10:59,902 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:10:59,902 [I] Checking connection
2026-09-24 00:10:59,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 00:11:00,157 [I] connection success
2026-09-24 00:11:00,158 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 00:11:00,158 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 00:11:00,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 00:11:00,439 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 00:11:00,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:11:00,439 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 00:11:00,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54751/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 00:11:00,742 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 00:11:01.219233 392 mini_ranger_kms.cc:310] Created kms service
I20260924 00:11:01.392140 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 00:11:01.434734 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 00:11:01 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208658, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:01.834362 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 00:11:01.834483 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 00:11:01.834506 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 00:11:01.837950 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11513
I20260924 00:11:01.838074 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54487
Sep 24, 2026 12:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:11:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:11:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:11:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 00:11:04.039064 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.218s user 0.004s sys 0.161s
I20260924 00:11:04.039172 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 00:11:04.039270 392 mini_ranger_kms.cc:401] 127.0.98.20:54487/kms/v1/keys
Sep 24 00:11:04 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790208664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 00:11:04 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:04 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:07 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:11:07.318280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33933
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54487/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:07.401672 11564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:07.401890 11564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:07.401912 11564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:07.403151 11564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:11:07.403192 11564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:07.403208 11564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:07.403285 11564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:11:07.403303 11564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:11:07.404856 11564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54487/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33933
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:07.405110 11564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:07.405361 11564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:11:07.410611 11564 server_base.cc:1034] running on GCE node
W20260924 00:11:07.410830 11570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:07.410823 11572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:07.411091 11569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:07.411368 11564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:07.411687 11564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:07.412856 11564 hybrid_clock.cc:648] HybridClock initialized: now 1790208667412842 us; error 32 us; skew 500 ppm
Sep 24 00:11:07 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:07.422421 11564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:11:07.423007 11564 webserver.cc:460] Webserver started at http://127.0.98.62:39585/ using document root <none> and password file <none>
I20260924 00:11:07.423353 11564 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:07.423418 11564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:07.423563 11564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:07 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:07.475525 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "342da430d10b410195ccace6cffd4131"
format_stamp: "Formatted at 2026-09-24 00:11:07 on dist-test-slave-3q8k"
server_key: "4ea295fc41a434cf048efcd106343233"
server_key_iv: "4f032315d7a41dde9b7de5c94a360712"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:07.475941 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "342da430d10b410195ccace6cffd4131"
format_stamp: "Formatted at 2026-09-24 00:11:07 on dist-test-slave-3q8k"
server_key: "4ea295fc41a434cf048efcd106343233"
server_key_iv: "4f032315d7a41dde9b7de5c94a360712"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:07.477416 11564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:11:07.489136 11579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:07.489374 11564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:11:07.489475 11564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "342da430d10b410195ccace6cffd4131"
format_stamp: "Formatted at 2026-09-24 00:11:07 on dist-test-slave-3q8k"
server_key: "4ea295fc41a434cf048efcd106343233"
server_key_iv: "4f032315d7a41dde9b7de5c94a360712"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:07.489604 11564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:07.517372 11564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:07.519668 11564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:07.519876 11564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:07.525264 11564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33933
I20260924 00:11:07.525290 11631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33933 every 8 connection(s)
I20260924 00:11:07.525681 11564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:11:07.526341 11632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:07.528698 11632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: Bootstrap starting.
I20260924 00:11:07.529313 11632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:07.529593 11632 log.cc:824] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:07.530347 11632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: No bootstrap required, opened a new log
I20260924 00:11:07.532862 11632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260924 00:11:07.533007 11632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:07.533030 11632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342da430d10b410195ccace6cffd4131, State: Initialized, Role: FOLLOWER
I20260924 00:11:07.533232 11632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260924 00:11:07.533299 11632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:11:07.533322 11632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:11:07.533390 11632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:07.533936 11632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260924 00:11:07.534020 11632 leader_election.cc:302] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342da430d10b410195ccace6cffd4131; no voters:
I20260924 00:11:07.534232 11632 leader_election.cc:288] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:11:07.534287 11635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:11:07.534477 11635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [term 1 LEADER]: Becoming Leader. State: Replica: 342da430d10b410195ccace6cffd4131, State: Running, Role: LEADER
I20260924 00:11:07.534492 11632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:11:07.534571 11635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } }
I20260924 00:11:07.534807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 11564
I20260924 00:11:07.534910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:11:07.534978 11635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342da430d10b410195ccace6cffd4131. Latest consensus state: current_term: 1 leader_uuid: "342da430d10b410195ccace6cffd4131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } } }
I20260924 00:11:07.535049 11635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:07.535105 392 external_mini_cluster.cc:1428] Setting key 6488bfd66b8e1ee52ea4d6fb2c1e1819
I20260924 00:11:07.535441 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342da430d10b410195ccace6cffd4131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342da430d10b410195ccace6cffd4131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33933 } } }
I20260924 00:11:07.535589 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:07.537254 11632 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:11:07.539644 11639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:11:07.540591 11639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 00:11:07 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:07.551793 11639 catalog_manager.cc:1347] Generated new cluster ID: 57b3333c43234c44a8a795b7ecd10639
I20260924 00:11:07.551865 11639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:11:07.551949 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:07.535729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.62:33933)
0924 00:11:07.535914 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:07.535924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:07.537770 (+ 1846us) server_negotiation.cc:407] Connection header received
0924 00:11:07.539392 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:07.539404 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:07.539551 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:07.539745 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:07.540435 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:07.542549 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:07.543661 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:07.544270 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:07.544410 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:07.547264 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:07.547269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:07.547269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:07.547289 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:07.550072 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:07.550508 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:07.550512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:07.550512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:07.550621 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:07.550854 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:07.550855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:07.550856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:07.551193 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:07.551413 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:07.551523 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:11:07.551655 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260924 00:11:07.573422 11639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:11:07.574180 11639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:11:07.579437 11639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: Generated new TSK 0
I20260924 00:11:07.579645 11639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:11:08.343251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:08.466596 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:08.466861 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:08.466900 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:08.468156 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:08.468232 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:08.468279 11677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:11:08.469854 11677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:08.470155 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:08.470424 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:08.475605 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.475605 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.475670 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:08.476249 11677 server_base.cc:1034] running on GCE node
I20260924 00:11:08.476466 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:08.476727 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:08.477931 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790208668477919 us; error 37 us; skew 500 ppm
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:08.482019 11677 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:11:08.482648 11677 webserver.cc:460] Webserver started at http://127.0.98.1:35033/ using document root <none> and password file <none>
I20260924 00:11:08.482956 11677 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:08.483031 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:08.483240 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:08.497246 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "682c80209eae4ad28f764233f4eefc7d"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f4db574c202c7e5b677fc616bdfc79a6"
server_key_iv: "d771d50d374bf26b60c5c1b77a36de5f"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.497735 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "682c80209eae4ad28f764233f4eefc7d"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f4db574c202c7e5b677fc616bdfc79a6"
server_key_iv: "d771d50d374bf26b60c5c1b77a36de5f"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.499893 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 00:11:08.513427 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.513716 11677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:11:08.513862 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "682c80209eae4ad28f764233f4eefc7d"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f4db574c202c7e5b677fc616bdfc79a6"
server_key_iv: "d771d50d374bf26b60c5c1b77a36de5f"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.513976 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:08.538805 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:08.541632 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:08.541992 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:08.542302 11677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:08.542733 11677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:08.542825 11677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.542943 11677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:08.543028 11677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.550714 11677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34695
I20260924 00:11:08.550781 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34695 every 8 connection(s)
I20260924 00:11:08.551177 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:08.560454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 11677
I20260924 00:11:08.560551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:11:08.560734 392 external_mini_cluster.cc:1428] Setting key def17d660a0654714d55ec3c97d6538c
I20260924 00:11:08.571480 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:08.553755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.1:54001)
0924 00:11:08.554302 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:08.554361 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:08.554963 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:08.555404 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:08.555408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:08.555653 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:08.555893 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:08.555899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.556641 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.556642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:08.557451 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:08.557453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.557880 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.557881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:08.558004 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:08.558558 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:08.558566 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:08.560145 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:08.569589 (+ 9444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:08.569599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:08.569602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:08.569966 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:08.570180 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:08.570181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:08.570181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:08.570324 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:08.570623 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:08.570629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:08.570827 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 00:11:08.570959 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 00:11:08.571151 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":77,"threads_started":1}
I20260924 00:11:08.571839 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:08.554047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54001 (local address 127.0.98.62:33933)
0924 00:11:08.555140 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:08.555142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:08.555150 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:11:08.555177 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:08.555177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:08.555189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:08.555268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:08.556013 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.556541 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.557561 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.557761 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.557861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:08.567379 (+ 9518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:08.567381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:08.567381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:08.567391 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:08.569458 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:08.570055 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:08.570057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:08.570057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:08.570117 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:08.570391 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:08.570392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:08.570392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:08.570543 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:08.570598 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:08.571645 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0924 00:11:08.571727 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":69,"threads_started":1}
I20260924 00:11:08.572382 11808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260924 00:11:08.572511 11808 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:08.572703 11808 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
I20260924 00:11:08.573287 11596 ts_manager.cc:194] Registered new tserver with Master: 682c80209eae4ad28f764233f4eefc7d (127.0.98.1:34695)
I20260924 00:11:08.574250 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54001
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:11:08.598268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:08.705986 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:08.706369 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:08.706508 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:08.707787 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:08.707854 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:08.707895 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:11:08.709342 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:08.709611 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:08.709888 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:08.715499 11822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.715499 11824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.715870 11821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:08.716076 11814 server_base.cc:1034] running on GCE node
I20260924 00:11:08.716312 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:08.716624 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:08.717806 11814 hybrid_clock.cc:648] HybridClock initialized: now 1790208668717772 us; error 28 us; skew 500 ppm
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:08.721014 11814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:11:08.721549 11814 webserver.cc:460] Webserver started at http://127.0.98.2:39605/ using document root <none> and password file <none>
I20260924 00:11:08.721797 11814 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:08.721880 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:08.722023 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:08.739274 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "01baed2ac07d46bebf887b25f77f2341"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "e503fa9d79da80cfa2a8b393a4572b50"
server_key_iv: "775a132ea9813415501cffd61aa30123"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.739777 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01baed2ac07d46bebf887b25f77f2341"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "e503fa9d79da80cfa2a8b393a4572b50"
server_key_iv: "775a132ea9813415501cffd61aa30123"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.743651 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 00:11:08.761198 11831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.761451 11814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 00:11:08.761543 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "01baed2ac07d46bebf887b25f77f2341"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "e503fa9d79da80cfa2a8b393a4572b50"
server_key_iv: "775a132ea9813415501cffd61aa30123"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:08.761638 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:08.778875 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:08.781481 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:08.781709 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:08.781968 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:08.782393 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:08.782476 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.782543 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:08.782593 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:08.789667 11814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37985
I20260924 00:11:08.790172 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:11:08.793440 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37985 every 8 connection(s)
I20260924 00:11:08.797804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 11814
I20260924 00:11:08.798039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:11:08.798290 392 external_mini_cluster.cc:1428] Setting key cf29d0b753f0aae5888299b98e7d017a
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:08.806460 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:08.794499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49207 (local address 127.0.98.62:33933)
0924 00:11:08.794562 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:08.794565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:08.794865 (+ 300us) server_negotiation.cc:407] Connection header received
0924 00:11:08.795601 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:08.795602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:08.795615 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:08.795645 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:08.796539 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.797310 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.799651 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.799992 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.800036 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:08.802770 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:08.802771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:08.802772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:08.802781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:08.804739 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:08.805310 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:08.805311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:08.805312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:08.805373 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:08.805660 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:08.805661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:08.805662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:08.805813 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:08.805856 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:08.806263 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 00:11:08.806358 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:11:08.807564 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:08.794215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.2:49207)
0924 00:11:08.794728 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:08.794778 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:08.795478 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:08.795849 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:08.795853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:08.796108 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:08.796429 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:08.796435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.797430 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.797431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:08.799549 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:08.799552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:08.800128 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:08.800130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:08.800280 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:08.800948 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:08.800959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:08.802632 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:08.804853 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:08.804861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:08.804864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:08.805214 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:08.805445 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:08.805446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:08.805446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:08.805585 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:08.805905 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:08.805911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:08.806098 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 00:11:08.806827 (+ 729us) client_negotiation.cc:240] Negotiation successful
0924 00:11:08.807009 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":84,"threads_started":1}
I20260924 00:11:08.808277 11945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260924 00:11:08.808374 11945 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:08.808562 11945 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
I20260924 00:11:08.809011 11596 ts_manager.cc:194] Registered new tserver with Master: 01baed2ac07d46bebf887b25f77f2341 (127.0.98.2:37985)
I20260924 00:11:08.809499 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49207
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:11:08.843297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:08.965503 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:08.965881 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:08.965977 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:08.967886 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:08.968113 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:08.968202 11951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:11:08.970520 11951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:08.971014 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:08.971374 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:08.978389 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.978853 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:08.979117 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:08.981025 11951 server_base.cc:1034] running on GCE node
I20260924 00:11:08.981254 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:08.981570 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:08.982777 11951 hybrid_clock.cc:648] HybridClock initialized: now 1790208668982753 us; error 37 us; skew 500 ppm
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:08.986191 11951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:11:08.986771 11951 webserver.cc:460] Webserver started at http://127.0.98.3:37117/ using document root <none> and password file <none>
I20260924 00:11:08.987084 11951 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:08.987191 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:08.987408 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:08 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:09.001550 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f4c497a72af4d15817e8702cbea2ee3"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f05391f5ec2602b659119487e2cce0d6"
server_key_iv: "4ed45ad7f15565230ede46cebd1ee0b4"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:09.002332 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f4c497a72af4d15817e8702cbea2ee3"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f05391f5ec2602b659119487e2cce0d6"
server_key_iv: "4ed45ad7f15565230ede46cebd1ee0b4"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:09.005455 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:11:09.024807 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:09.025146 11951 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 00:11:09.025276 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "1f4c497a72af4d15817e8702cbea2ee3"
format_stamp: "Formatted at 2026-09-24 00:11:08 on dist-test-slave-3q8k"
server_key: "f05391f5ec2602b659119487e2cce0d6"
server_key_iv: "4ed45ad7f15565230ede46cebd1ee0b4"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:09.025395 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:09.046452 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:09.051596 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:09.051816 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:09.052079 11951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:09.052454 11951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:09.052526 11951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:09.052582 11951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:09.052626 11951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:09.060174 11951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42721
I20260924 00:11:09.060307 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42721 every 8 connection(s)
I20260924 00:11:09.060775 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:11:09.070885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 11951
I20260924 00:11:09.070996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:11:09.071175 392 external_mini_cluster.cc:1428] Setting key da79bbdfc60c289c733bbeadc8e6cafc
Sep 24 00:11:09 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:09.082459 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.072142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35039 (local address 127.0.98.62:33933)
0924 00:11:09.072184 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.072186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.072511 (+ 325us) server_negotiation.cc:407] Connection header received
0924 00:11:09.073172 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.073174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.073186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.073216 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:09.074043 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.074756 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.076269 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.076553 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.076592 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.079243 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:09.079245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:09.079245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:09.079253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:09.080920 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:09.081398 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:09.081399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:09.081399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:09.081455 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:09.081736 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:09.081737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:09.081738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:09.081863 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:09.081903 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.082261 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.082362 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:11:09.083827 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.072178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33933 (local address 127.0.98.3:35039)
0924 00:11:09.072388 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.072433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.073050 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.073408 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.073413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.073655 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:09.073939 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.073944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.074873 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.074874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.076170 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.076174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.076690 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.076690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.076844 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.077487 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:09.077497 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:09.079113 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:09.081017 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:09.081023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:09.081025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:09.081314 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:09.081532 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:09.081533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:09.081533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:09.081657 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:09.081943 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:09.081948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:09.082117 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.083406 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.083574 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":136,"threads_started":1}
I20260924 00:11:09.084424 12080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33933
I20260924 00:11:09.084514 12080 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:09.084707 12080 heartbeater.cc:511] Master 127.0.98.62:33933 requested a full tablet report, sending...
I20260924 00:11:09.085099 11596 ts_manager.cc:194] Registered new tserver with Master: 1f4c497a72af4d15817e8702cbea2ee3 (127.0.98.3:42721)
I20260924 00:11:09.085557 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35039
I20260924 00:11:09.094126 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:11:09.102713 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.095970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.62:33933)
0924 00:11:09.096017 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.096020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.096372 (+ 352us) server_negotiation.cc:407] Connection header received
0924 00:11:09.096449 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.096450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.096462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.096516 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:09.097114 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.097611 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.098487 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.098700 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.098782 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.100089 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:09.100091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:09.100091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:09.100098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:09.101562 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:09.101939 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:09.101940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:09.101940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:09.101996 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:09.102196 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:09.102197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:09.102198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:09.102311 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:09.102419 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.102521 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.102596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:11:09.104543 11596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:11:09.106173 11596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:11:09.114647 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.109316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.3:42721)
0924 00:11:09.109723 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.109727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.109736 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:11:09.109779 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.109782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.109795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.109894 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:09.110139 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.110824 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.113350 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.114096 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.114162 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.114236 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.114512 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.114552 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":71,"threads_started":1}
I20260924 00:11:09.114636 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.108848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42721 (local address 127.0.0.1:48498)
0924 00:11:09.109557 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.109567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.109609 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.109903 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.109906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.109909 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:09.109986 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.109988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.110916 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.110917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.112526 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.112528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.114327 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.114328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.114417 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.114435 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.114497 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.114547 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":62,"threads_started":1}
I20260924 00:11:09.115033 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.109014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34695 (local address 127.0.0.1:33362)
0924 00:11:09.109990 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.110000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.110044 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.111126 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.111127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.111129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:09.111169 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.111171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.112416 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.112417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.113233 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.113235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.114843 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.114844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.114909 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.114913 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.114940 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.114975 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":72,"threads_started":1}
I20260924 00:11:09.115381 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.109308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.98.1:34695)
0924 00:11:09.109450 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.109455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.110940 (+ 1485us) server_negotiation.cc:407] Connection header received
0924 00:11:09.110978 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.110980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.110997 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.111065 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:09.111264 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.112328 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.113308 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.114750 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.115123 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.115208 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.115235 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.115315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260924 00:11:09.115986 11742 tablet_service.cc:1467] Processing CreateTablet for tablet 2005650c3edb402d93c0a98e21669749 (DEFAULT_TABLE table=test-table [id=1161528467784ef7845fa5a9d8c8d7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:09.116299 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005650c3edb402d93c0a98e21669749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:09.119130 12098 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Bootstrap starting.
I20260924 00:11:09.119704 12014 tablet_service.cc:1467] Processing CreateTablet for tablet 2005650c3edb402d93c0a98e21669749 (DEFAULT_TABLE table=test-table [id=1161528467784ef7845fa5a9d8c8d7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:09.120065 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005650c3edb402d93c0a98e21669749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:09.120342 12098 tablet_bootstrap.cc:654] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:09.120630 12098 log.cc:824] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:09.123181 12100 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Bootstrap starting.
I20260924 00:11:09.124555 12100 tablet_bootstrap.cc:654] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:09.124857 12100 log.cc:824] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:09.135780 12100 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: No bootstrap required, opened a new log
I20260924 00:11:09.135892 12100 ts_tablet_manager.cc:1397] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 00:11:09.137991 12100 raft_consensus.cc:348] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.138180 12100 raft_consensus.cc:374] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:09.138226 12100 raft_consensus.cc:729] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4c497a72af4d15817e8702cbea2ee3, State: Initialized, Role: FOLLOWER
I20260924 00:11:09.138608 12100 consensus_queue.cc:260] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.138931 12100 ts_tablet_manager.cc:1428] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:11:09.139089 12080 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
I20260924 00:11:09.142746 12098 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: No bootstrap required, opened a new log
I20260924 00:11:09.142824 12098 ts_tablet_manager.cc:1397] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20260924 00:11:09.144791 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.108603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37985 (local address 127.0.0.1:37798)
0924 00:11:09.108992 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.109034 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.109099 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.123934 (+ 14835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.123935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.123939 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:09.124001 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.124004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.142055 (+ 18051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.142056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.143287 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.143292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.144558 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.144560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.144626 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.144631 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.144663 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.144711 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":80,"threads_started":1}
I20260924 00:11:09.145396 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.108705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.98.2:37985)
0924 00:11:09.114336 (+ 5631us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.114341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.114350 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:11:09.114387 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.114389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.114405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.124098 (+ 9693us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:09.124189 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.141955 (+ 17766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.143603 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.144447 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.145144 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.145245 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.145273 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.145315 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5586,"thread_start_us":89,"threads_started":1}
I20260924 00:11:09.146193 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 2005650c3edb402d93c0a98e21669749 (DEFAULT_TABLE table=test-table [id=1161528467784ef7845fa5a9d8c8d7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:09.146575 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005650c3edb402d93c0a98e21669749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:09.150194 12098 raft_consensus.cc:348] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.150429 12098 raft_consensus.cc:374] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:09.150480 12098 raft_consensus.cc:729] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682c80209eae4ad28f764233f4eefc7d, State: Initialized, Role: FOLLOWER
I20260924 00:11:09.150636 12098 consensus_queue.cc:260] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.150916 12098 ts_tablet_manager.cc:1428] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260924 00:11:09.151042 11808 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
I20260924 00:11:09.152696 12103 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Bootstrap starting.
I20260924 00:11:09.153705 12103 tablet_bootstrap.cc:654] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:09.154001 12103 log.cc:824] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:09.154728 12103 tablet_bootstrap.cc:492] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: No bootstrap required, opened a new log
I20260924 00:11:09.154804 12103 ts_tablet_manager.cc:1397] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:11:09.156654 12103 raft_consensus.cc:348] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.156823 12103 raft_consensus.cc:374] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:09.156867 12103 raft_consensus.cc:729] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01baed2ac07d46bebf887b25f77f2341, State: Initialized, Role: FOLLOWER
I20260924 00:11:09.156992 12103 consensus_queue.cc:260] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.157285 12103 ts_tablet_manager.cc:1428] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:11:09.157294 11945 heartbeater.cc:503] Master 127.0.98.62:33933 was elected leader, sending a full tablet report...
Sep 24 00:11:09 dist-test-slave-3q8k krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 00:11:09.295059 11946 tablet.cc:2369] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:11:09.301002 12106 raft_consensus.cc:482] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:11:09.301167 12106 raft_consensus.cc:504] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.301638 12106 leader_election.cc:288] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4c497a72af4d15817e8702cbea2ee3 (127.0.98.3:42721), 682c80209eae4ad28f764233f4eefc7d (127.0.98.1:34695)
W20260924 00:11:09.304725 11809 tablet.cc:2369] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:11:09.305481 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.301795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42721 (local address 127.0.98.2:37319)
0924 00:11:09.301921 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.301937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.301985 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.302289 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.302290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.302292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:09.302365 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.302367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.303293 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.303294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.304521 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.304523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.305238 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.305238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.305320 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.305323 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.305370 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.305407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 00:11:09.305760 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.302067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37319 (local address 127.0.98.3:42721)
0924 00:11:09.302113 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.302115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.302126 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:11:09.302157 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.302157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.302167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.302220 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:09.302439 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.303137 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.304633 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.305139 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.305199 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.305211 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.305655 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.305695 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:11:09.306133 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005650c3edb402d93c0a98e21669749" candidate_uuid: "01baed2ac07d46bebf887b25f77f2341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4c497a72af4d15817e8702cbea2ee3" is_pre_election: true
I20260924 00:11:09.306284 12034 raft_consensus.cc:2454] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01baed2ac07d46bebf887b25f77f2341 in term 0.
I20260924 00:11:09.306584 11833 leader_election.cc:302] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01baed2ac07d46bebf887b25f77f2341, 1f4c497a72af4d15817e8702cbea2ee3; no voters:
I20260924 00:11:09.306782 12106 raft_consensus.cc:2781] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:11:09.306834 12106 raft_consensus.cc:482] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:11:09.306855 12106 raft_consensus.cc:3037] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:09.307744 12106 raft_consensus.cc:504] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.307881 12106 leader_election.cc:288] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4c497a72af4d15817e8702cbea2ee3 (127.0.98.3:42721), 682c80209eae4ad28f764233f4eefc7d (127.0.98.1:34695)
I20260924 00:11:09.308197 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005650c3edb402d93c0a98e21669749" candidate_uuid: "01baed2ac07d46bebf887b25f77f2341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4c497a72af4d15817e8702cbea2ee3"
I20260924 00:11:09.308274 12034 raft_consensus.cc:3037] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:09.309154 12034 raft_consensus.cc:2454] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01baed2ac07d46bebf887b25f77f2341 in term 1.
I20260924 00:11:09.309405 11833 leader_election.cc:302] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01baed2ac07d46bebf887b25f77f2341, 1f4c497a72af4d15817e8702cbea2ee3; no voters:
I20260924 00:11:09.309520 12106 raft_consensus.cc:2781] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:11:09.309672 12106 raft_consensus.cc:686] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 1 LEADER]: Becoming Leader. State: Replica: 01baed2ac07d46bebf887b25f77f2341, State: Running, Role: LEADER
I20260924 00:11:09.309779 12106 consensus_queue.cc:237] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } }
I20260924 00:11:09.310580 11596 catalog_manager.cc:5495] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01baed2ac07d46bebf887b25f77f2341 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "01baed2ac07d46bebf887b25f77f2341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01baed2ac07d46bebf887b25f77f2341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 } health_report { overall_health: UNKNOWN } } }
I20260924 00:11:09.312947 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.301989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43531 (local address 127.0.98.1:34695)
0924 00:11:09.302122 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.302124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.302562 (+ 438us) server_negotiation.cc:407] Connection header received
0924 00:11:09.303601 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.303603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.303616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.303646 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:09.303862 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.304536 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.311834 (+ 7298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.312460 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.312512 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.312523 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.312826 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.312876 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 00:11:09.313328 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.301807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34695 (local address 127.0.98.2:43531)
0924 00:11:09.302227 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.302238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:09.303293 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:09.303710 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:09.303710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:09.303712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:09.303789 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.303790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.305081 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.305082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.311709 (+ 6627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:09.311714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.312607 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.312608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:09.312731 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.312735 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:11:09.313179 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 00:11:09.313229 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":93,"threads_started":1}
I20260924 00:11:09.313757 11762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005650c3edb402d93c0a98e21669749" candidate_uuid: "01baed2ac07d46bebf887b25f77f2341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682c80209eae4ad28f764233f4eefc7d" is_pre_election: true
I20260924 00:11:09.313910 11762 raft_consensus.cc:2454] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01baed2ac07d46bebf887b25f77f2341 in term 0.
I20260924 00:11:09.314044 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005650c3edb402d93c0a98e21669749" candidate_uuid: "01baed2ac07d46bebf887b25f77f2341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682c80209eae4ad28f764233f4eefc7d"
I20260924 00:11:09.314116 11761 raft_consensus.cc:3037] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:09.314885 11761 raft_consensus.cc:2454] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01baed2ac07d46bebf887b25f77f2341 in term 1.
W20260924 00:11:09.322780 12081 tablet.cc:2369] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:11:09.351434 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.347853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.0.98.2:37985)
0924 00:11:09.347911 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:09.347913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:09.347925 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:11:09.347976 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:09.347977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:09.348006 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:09.348106 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:11:09.348346 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.349088 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.350226 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:09.350530 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:09.350599 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:09.350820 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:11:09.350964 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:11:09.351047 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:09.351324 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 00:11:09.351346 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:11:09.364571 11761 raft_consensus.cc:1261] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 1 FOLLOWER]: Refusing update from remote peer 01baed2ac07d46bebf887b25f77f2341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:11:09.365118 12108 consensus_queue.cc:1035] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682c80209eae4ad28f764233f4eefc7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:11:09.367656 12034 raft_consensus.cc:1261] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 1 FOLLOWER]: Refusing update from remote peer 01baed2ac07d46bebf887b25f77f2341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:11:09.368036 12108 consensus_queue.cc:1035] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4c497a72af4d15817e8702cbea2ee3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:11:09.377640 12112 mvcc.cc:205] Tried to move back new op lower bound from 7332694709713436672 to 7332694709493370880. Current Snapshot: MvccSnapshot[applied={T|T < 7332694709713436672}]
I20260924 00:11:09.378575 12114 mvcc.cc:205] Tried to move back new op lower bound from 7332694709713436672 to 7332694709493370880. Current Snapshot: MvccSnapshot[applied={T|T < 7332694709713436672}]
I20260924 00:11:09.381959 11596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:11:09.382058 11596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:11:09.383006 11596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 342da430d10b410195ccace6cffd4131: Sending DeleteTablet for 3 replicas of tablet 2005650c3edb402d93c0a98e21669749
I20260924 00:11:09.383397 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet 2005650c3edb402d93c0a98e21669749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48498
I20260924 00:11:09.383451 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2005650c3edb402d93c0a98e21669749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33362
I20260924 00:11:09.383720 12120 tablet_replica.cc:330] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: stopping tablet replica
I20260924 00:11:09.383803 11879 tablet_service.cc:1514] Processing DeleteTablet for tablet 2005650c3edb402d93c0a98e21669749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37798
I20260924 00:11:09.383853 12121 tablet_replica.cc:330] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d: stopping tablet replica
I20260924 00:11:09.383956 12121 raft_consensus.cc:2229] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:11:09.384073 12121 raft_consensus.cc:2258] T 2005650c3edb402d93c0a98e21669749 P 682c80209eae4ad28f764233f4eefc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:11:09.384148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 11677
I20260924 00:11:09.384253 12120 raft_consensus.cc:2229] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:11:09.384389 12120 raft_consensus.cc:2258] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:11:09.384995 12120 ts_tablet_manager.cc:1905] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:11:09.386842 12120 ts_tablet_manager.cc:1918] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:11:09.387032 12120 log.cc:1196] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/2005650c3edb402d93c0a98e21669749
I20260924 00:11:09.387598 12120 ts_tablet_manager.cc:1939] T 2005650c3edb402d93c0a98e21669749 P 1f4c497a72af4d15817e8702cbea2ee3: Deleting consensus metadata
I20260924 00:11:09.388046 11581 catalog_manager.cc:4841] TS 1f4c497a72af4d15817e8702cbea2ee3 (127.0.98.3:42721): tablet 2005650c3edb402d93c0a98e21669749 (table test-table [id=1161528467784ef7845fa5a9d8c8d7bc]) successfully deleted
I20260924 00:11:09.391396 12122 tablet_replica.cc:330] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: stopping tablet replica
I20260924 00:11:09.391544 12122 raft_consensus.cc:2229] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:11:09.391748 12122 raft_consensus.cc:2258] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:11:09.392336 12122 ts_tablet_manager.cc:1905] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:11:09.394775 12122 ts_tablet_manager.cc:1918] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:11:09.394899 12122 log.cc:1196] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/2005650c3edb402d93c0a98e21669749
I20260924 00:11:09.395366 12122 ts_tablet_manager.cc:1939] T 2005650c3edb402d93c0a98e21669749 P 01baed2ac07d46bebf887b25f77f2341: Deleting consensus metadata
W20260924 00:11:09.395750 11582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34695 (error 108)
I20260924 00:11:09.396669 11580 catalog_manager.cc:4841] TS 01baed2ac07d46bebf887b25f77f2341 (127.0.98.2:37985): tablet 2005650c3edb402d93c0a98e21669749 (table test-table [id=1161528467784ef7845fa5a9d8c8d7bc]) successfully deleted
I20260924 00:11:09.396951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 11814
I20260924 00:11:09.399605 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:09.399484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34695 (local address 127.0.0.1:33374)
0924 00:11:09.399528 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:09.399567 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 00:11:09.399775 11582 catalog_manager.cc:4584] TS 682c80209eae4ad28f764233f4eefc7d (127.0.98.1:34695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2005650c3edb402d93c0a98e21669749: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34695: connect: Connection refused (error 111)
I20260924 00:11:09.401937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 11951
I20260924 00:11:09.406879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 11564
I20260924 00:11:09.412109 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 00:11:09.756697 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 00:11:09.756796 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:11:10.101626 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:11:10.101 UTC [11150] LOG: received smart shutdown request
2026-09-24 00:11:10.106 UTC [11150] LOG: background worker "logical replication launcher" (PID 11159) exited with exit code 1
2026-09-24 00:11:10.106 UTC [11154] LOG: shutting down
2026-09-24 00:11:10.304 UTC [11150] LOG: database system is shut down
2026-09-24T00:11:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46625 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:11:10 dist-test-slave-3q8k krb5kdc[12132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:11:10 dist-test-slave-3q8k krb5kdc[12132](info): set up 2 sockets
Sep 24 00:11:10 dist-test-slave-3q8k krb5kdc[12132](info): commencing operation
krb5kdc: starting...
W20260924 00:11:11.483326 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:11:11 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:11:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 00:11:11.648351 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskElJ0hL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/postgres -l logfile start
2026-09-24 00:11:14.138 UTC [12162] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:11:14.138 UTC [12162] LOG: listening on IPv4 address "127.0.98.20", port 43467
2026-09-24 00:11:14.142 UTC [12162] LOG: listening on Unix socket "/tmp/.s.PGSQL.43467"
2026-09-24 00:11:14.177 UTC [12165] LOG: database system was shut down at 2026-09-24 00:11:12 UTC
2026-09-24 00:11:14.181 UTC [12162] LOG: database system is ready to accept connections
I20260924 00:11:15.135067 392 mini_postgres.cc:96] Postgres bound to 43467
2026-09-24 00:11:15.140 UTC [12173] FATAL: database "slave" does not exist
127.0.98.20:43467 - accepting connections
I20260924 00:11:15.141345 392 mini_ranger.cc:161] Starting Ranger...
I20260924 00:11:15.156064 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:11:15.626106 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:11:15.626173 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin
2026-09-24 00:11:15,912 [I] DB FLAVOR :POSTGRES
2026-09-24 00:11:15,913 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:11:15,913 [I] Checking connection..
2026-09-24 00:11:15,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:11:16,178 [I] Checking connection passed.
2026-09-24 00:11:16,178 [I] --------- Verifying version history table ---------
2026-09-24 00:11:16,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:11:16,418 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:11:16,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:11:16,651 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:11:16,651 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:11:16,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:11:16.878 UTC [12237] WARNING: there is no transaction in progress
2026-09-24 00:11:16,885 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:11:16,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:11:17,148 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:11:17,149 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:11:17,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:11:17,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:11:17,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3q8k.c.gcp-upstream.internal','N') ;"
2026-09-24 00:11:17,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:11:18,149 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:11:18,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:11:18,411 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:11:18,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:11:18,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:11:18,869 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:11:18,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:11:19.843 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.859 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.877 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.893 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.909 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.925 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.979 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.986 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:19.999 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.009 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.019 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.033 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.043 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.049 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.056 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20.063 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:11:20,719 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:11:20,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:11:20,974 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:11:20,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:11:21,207 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:11:21,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:11:21,442 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:11:21,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:11:21,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:11:21,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3q8k.c.gcp-upstream.internal';"
2026-09-24 00:11:22,164 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:11:22,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:11:22,402 [I] DB_PATCHES have already been applied
I20260924 00:11:22.408085 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskElJ0hL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:11:22.408174 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 00:11:22.411099 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39315
Sep 24, 2026 12:11:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:11:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:11:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:11:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:11:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:11:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:11:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 00:11:23.548277 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.407s user 0.001s sys 0.007s
Sep 24, 2026 12:11:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:11:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 12:11:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:11:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:11:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:11:45.267891 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 00:11:45 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208705, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 00:11:45.351817 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 00:11:45.361426 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 00:11:46.239451 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 00:11:46.239522 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms
I20260924 00:11:46.239537 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43467
2026-09-24 00:11:46,414 [I] DB FLAVOR :POSTGRES
2026-09-24 00:11:46,414 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:11:46,414 [I] Checking connection
2026-09-24 00:11:46,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 00:11:46,667 [I] connection success
2026-09-24 00:11:46,668 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 00:11:46,668 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 00:11:46,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 00:11:46,923 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 00:11:46,923 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:11:46,923 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 00:11:46,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 00:11:47,212 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 00:11:47.658788 392 mini_ranger_kms.cc:310] Created kms service
I20260924 00:11:47.841526 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 00:11:47.886152 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 00:11:47 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208705, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:48.288439 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 00:11:48.288659 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskElJ0hL/build/release/bin/postgresql.jar:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskElJ0hL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 00:11:48.288683 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 00:11:48.292299 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12525
I20260924 00:11:48.292388 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54815
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:11:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 00:11:50.561340 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.210s user 0.007s sys 0.150s
I20260924 00:11:50.561430 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 00:11:50.561475 392 mini_ranger_kms.cc:401] 127.0.98.20:54815/kms/v1/keys
Sep 24 00:11:50 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790208710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 00:11:51 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:51 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208710, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:53 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:11:53.790414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:53.897539 12576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:53.897796 12576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:53.897835 12576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:53.899015 12576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:11:53.899087 12576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:53.899111 12576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:53.899130 12576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:11:53.899161 12576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:11:53.900870 12576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:53.901160 12576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:53.901410 12576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:53.906783 12582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:53.906832 12581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:53.906936 12576 server_base.cc:1034] running on GCE node
W20260924 00:11:53.907038 12584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:53.907341 12576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:53.907696 12576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:53.908852 12576 hybrid_clock.cc:648] HybridClock initialized: now 1790208713908828 us; error 41 us; skew 500 ppm
Sep 24 00:11:53 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:53.917959 12576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:11:53.918534 12576 webserver.cc:460] Webserver started at http://127.0.98.62:43437/ using document root <none> and password file <none>
I20260924 00:11:53.918793 12576 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:53.918835 12576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:53.918982 12576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:53 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:53.965130 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "17552fe783b444d3ad7de71f0e613a94"
format_stamp: "Formatted at 2026-09-24 00:11:53 on dist-test-slave-3q8k"
server_key: "b58e6bdf8e55a358a5ab2b722c055c57"
server_key_iv: "73cc39a00f42e4ecddabda2824c65513"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:53.965526 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "17552fe783b444d3ad7de71f0e613a94"
format_stamp: "Formatted at 2026-09-24 00:11:53 on dist-test-slave-3q8k"
server_key: "b58e6bdf8e55a358a5ab2b722c055c57"
server_key_iv: "73cc39a00f42e4ecddabda2824c65513"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:53.967070 12576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:11:53.979735 12591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:53.981256 12576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 00:11:53.981406 12576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "17552fe783b444d3ad7de71f0e613a94"
format_stamp: "Formatted at 2026-09-24 00:11:53 on dist-test-slave-3q8k"
server_key: "b58e6bdf8e55a358a5ab2b722c055c57"
server_key_iv: "73cc39a00f42e4ecddabda2824c65513"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:53.981513 12576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:53.989645 12576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:53.991685 12576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:53.991833 12576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:53.996935 12576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39863
I20260924 00:11:53.996961 12643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39863 every 8 connection(s)
I20260924 00:11:53.997438 12576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:11:53.998162 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:54.000595 12644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Bootstrap starting.
I20260924 00:11:54.001274 12644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:54.001586 12644 log.cc:824] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:54.002307 12644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: No bootstrap required, opened a new log
I20260924 00:11:54.004592 12644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:54.004756 12644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:54.004781 12644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17552fe783b444d3ad7de71f0e613a94, State: Initialized, Role: FOLLOWER
I20260924 00:11:54.004951 12644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:54.005036 12644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:11:54.005060 12644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:11:54.005128 12644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:54.005744 12644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:54.005836 12644 leader_election.cc:302] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17552fe783b444d3ad7de71f0e613a94; no voters:
I20260924 00:11:54.006081 12644 leader_election.cc:288] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:11:54.006105 12647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:11:54.006225 12647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 LEADER]: Becoming Leader. State: Replica: 17552fe783b444d3ad7de71f0e613a94, State: Running, Role: LEADER
I20260924 00:11:54.006309 12647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:54.006368 12644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:11:54.006639 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17552fe783b444d3ad7de71f0e613a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 00:11:54.006796 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:54.007038 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17552fe783b444d3ad7de71f0e613a94. Latest consensus state: current_term: 1 leader_uuid: "17552fe783b444d3ad7de71f0e613a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 00:11:54.007159 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:54.007525 12654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:11:54.007552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 12576
I20260924 00:11:54.007651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:11:54.007827 392 external_mini_cluster.cc:1428] Setting key 9fa441f5a47f89728f810158062f767d
I20260924 00:11:54.008065 12654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:11:54.008888 12644 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:11:54.011370 12654 catalog_manager.cc:1347] Generated new cluster ID: a5aef91b4ce54127ba4d0c0d72a61c1f
I20260924 00:11:54.011462 12654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:11:54 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:54.020848 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:54.008490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.98.62:39863)
0924 00:11:54.008665 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:54.008676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:54.008711 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:11:54.008840 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:54.008849 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:54.008914 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:54.009897 (+ 983us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:54.010026 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.011483 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.012572 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.013159 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.016417 (+ 3258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:54.016518 (+ 101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:54.016522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:54.016522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:54.016532 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:54.018606 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:54.019022 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:54.019025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:54.019025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:54.019133 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:54.019397 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:54.019398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:54.019398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:54.019749 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:54.020403 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:54.020481 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:11:54.020614 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260924 00:11:54.024657 12654 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:11:54.026084 12654 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:11:54.030819 12654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Generated new TSK 0
I20260924 00:11:54.031059 12654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:11:54.790771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:54.907998 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:54.908275 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:54.908314 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:54.910075 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:54.910141 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:54.910184 12689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:11:54.912439 12689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:54.912703 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:54.912993 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:54.918656 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:54.918663 12697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:54.919031 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:54.919317 12689 server_base.cc:1034] running on GCE node
I20260924 00:11:54.919516 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:54.919770 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:54.920946 12689 hybrid_clock.cc:648] HybridClock initialized: now 1790208714920915 us; error 44 us; skew 500 ppm
Sep 24 00:11:54 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:54.924453 12689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:11:54.925032 12689 webserver.cc:460] Webserver started at http://127.0.98.1:45537/ using document root <none> and password file <none>
I20260924 00:11:54.925293 12689 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:54.925428 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:54.925601 12689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:54 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:54.940183 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "df60bbcc5e7e440abbf1872f21041b68"
format_stamp: "Formatted at 2026-09-24 00:11:54 on dist-test-slave-3q8k"
server_key: "698498e51d4e0ac2a63695127ef48536"
server_key_iv: "a711a3512c0db8b3e5d0057fe2aef5f5"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:54.940689 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df60bbcc5e7e440abbf1872f21041b68"
format_stamp: "Formatted at 2026-09-24 00:11:54 on dist-test-slave-3q8k"
server_key: "698498e51d4e0ac2a63695127ef48536"
server_key_iv: "a711a3512c0db8b3e5d0057fe2aef5f5"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:54.943619 12689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:11:54.957695 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:54.957983 12689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 00:11:54.958092 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "df60bbcc5e7e440abbf1872f21041b68"
format_stamp: "Formatted at 2026-09-24 00:11:54 on dist-test-slave-3q8k"
server_key: "698498e51d4e0ac2a63695127ef48536"
server_key_iv: "a711a3512c0db8b3e5d0057fe2aef5f5"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:54.958180 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:54.974084 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:54.976632 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:54.976815 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:54.977059 12689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:54.977474 12689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:54.977571 12689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:54.977669 12689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:54.977739 12689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:54.984984 12689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36091
I20260924 00:11:54.985522 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:11:54.987087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 12689
I20260924 00:11:54.987197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:11:54.987422 392 external_mini_cluster.cc:1428] Setting key 43aeb2cf376420e88c1cbf3854deaf1c
I20260924 00:11:54.987145 12819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36091 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 00:11:54 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:55.006757 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:54.987973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:49255)
0924 00:11:54.993282 (+ 5309us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:54.993336 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:54.994148 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:54.994574 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:54.994578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:54.995033 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:54.995347 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:54.995354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.996546 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.996547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:54.997822 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:54.997825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.998578 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.998579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:54.998731 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:54.999485 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:54.999497 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:55.002872 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:55.005040 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.005048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.005051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.005382 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.005608 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.005609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.005609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.005750 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.006051 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:55.006057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:55.006240 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 00:11:55.006353 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 00:11:55.006513 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5259,"thread_start_us":82,"threads_started":1}
I20260924 00:11:55.007114 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:54.988335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49255 (local address 127.0.98.62:39863)
0924 00:11:54.988485 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:54.988487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:54.993562 (+ 5075us) server_negotiation.cc:407] Connection header received
0924 00:11:54.994284 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:54.994286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:54.994301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:54.994354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:54.995650 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.996427 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.997929 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:54.998279 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:54.998364 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.002995 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:55.002996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:55.002997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:55.003005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:55.004933 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.005473 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.005474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.005474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.005541 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.005824 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.005825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.005825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.005967 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:55.006025 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:55.006925 (+ 900us) server_negotiation.cc:299] Negotiation successful
0924 00:11:55.007013 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260924 00:11:55.007531 12820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:55.007630 12820 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.007827 12820 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:55.008338 12608 ts_manager.cc:194] Registered new tserver with Master: df60bbcc5e7e440abbf1872f21041b68 (127.0.98.1:36091)
I20260924 00:11:55.009193 12608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:11:55.026710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:55.133440 12826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:55.133677 12826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:55.133716 12826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:55.135036 12826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:55.135111 12826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:55.135157 12826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:11:55.136710 12826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:55.136978 12826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:55.137228 12826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:55.142405 12836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.142441 12834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.143054 12833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.143191 12826 server_base.cc:1034] running on GCE node
I20260924 00:11:55.143482 12826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:55.143760 12826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:55.144936 12826 hybrid_clock.cc:648] HybridClock initialized: now 1790208715144899 us; error 42 us; skew 500 ppm
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:55.147555 12826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:11:55.148101 12826 webserver.cc:460] Webserver started at http://127.0.98.2:44119/ using document root <none> and password file <none>
I20260924 00:11:55.148350 12826 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:55.148407 12826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:55.148533 12826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:55.162191 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "20ce4e3edbb2446c9904ad4750f1b5a0"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "ef36c905f0ec5ee32875cdff59f07a01"
server_key_iv: "468b5cd1b287744a3d0ac10a62bd77d8"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.162770 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20ce4e3edbb2446c9904ad4750f1b5a0"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "ef36c905f0ec5ee32875cdff59f07a01"
server_key_iv: "468b5cd1b287744a3d0ac10a62bd77d8"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.167534 12826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 00:11:55.199385 12843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.199620 12826 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.001s sys 0.000s
I20260924 00:11:55.199729 12826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "20ce4e3edbb2446c9904ad4750f1b5a0"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "ef36c905f0ec5ee32875cdff59f07a01"
server_key_iv: "468b5cd1b287744a3d0ac10a62bd77d8"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.199867 12826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:55.219266 12826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:55.221715 12826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:55.221854 12826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:55.222090 12826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:55.222436 12826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:55.222477 12826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.222506 12826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:55.222524 12826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.229753 12826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33707
I20260924 00:11:55.229909 12956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33707 every 8 connection(s)
I20260924 00:11:55.230212 12826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:11:55.235534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 12826
I20260924 00:11:55.235643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:11:55.235813 392 external_mini_cluster.cc:1428] Setting key c51ce32fdac674c9025fe7d573da502b
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:55.242538 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.231359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49419 (local address 127.0.98.62:39863)
0924 00:11:55.231424 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:55.231427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:55.231941 (+ 514us) server_negotiation.cc:407] Connection header received
0924 00:11:55.232669 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:55.232670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:55.232688 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:55.232721 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:55.233603 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.234372 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.235936 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.236229 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.236276 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.238924 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:55.238926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:55.238927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:55.238935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:55.240769 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.241301 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.241302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.241303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.241377 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.241719 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.241720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.241721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.241866 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:55.241913 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:55.242331 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 00:11:55.242441 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:11:55.243108 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.231534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:49419)
0924 00:11:55.231800 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:55.231845 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:55.232528 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:55.232912 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:55.232917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:55.233169 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:55.233485 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.233493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.234514 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.234515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.235826 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.235829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.236384 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.236385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.236540 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.237184 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:55.237194 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:55.238807 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:55.240885 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.240891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.240894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.241204 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.241483 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.241484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.241485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.241632 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.241966 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:55.241972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:55.242164 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 00:11:55.242680 (+ 516us) client_negotiation.cc:240] Negotiation successful
0924 00:11:55.242856 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260924 00:11:55.243631 12957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:55.243734 12957 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.243935 12957 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:55.244293 12608 ts_manager.cc:194] Registered new tserver with Master: 20ce4e3edbb2446c9904ad4750f1b5a0 (127.0.98.2:33707)
I20260924 00:11:55.244805 12608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49419
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:11:55.275475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:55.387058 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:55.387441 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:55.387529 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:55.389338 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:55.389456 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:55.389541 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:11:55.391809 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:55.392313 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:55.392637 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:55.399554 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.399891 12963 server_base.cc:1034] running on GCE node
W20260924 00:11:55.400134 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.400620 12971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.400882 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:55.401160 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:55.402333 12963 hybrid_clock.cc:648] HybridClock initialized: now 1790208715402307 us; error 40 us; skew 500 ppm
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:55.405616 12963 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:11:55.406158 12963 webserver.cc:460] Webserver started at http://127.0.98.3:41639/ using document root <none> and password file <none>
I20260924 00:11:55.406441 12963 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:55.406509 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:55.406651 12963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:55.424533 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "73a09d99f1c94fb4bf368cf60354d561"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "5fc6eb4f8785f3473c939fc382769c90"
server_key_iv: "e428b8113968f5e21aedd1673b23e8c0"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.425019 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73a09d99f1c94fb4bf368cf60354d561"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "5fc6eb4f8785f3473c939fc382769c90"
server_key_iv: "e428b8113968f5e21aedd1673b23e8c0"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.426980 12963 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 00:11:55.441578 12978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.441763 12963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 00:11:55.441854 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "73a09d99f1c94fb4bf368cf60354d561"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "5fc6eb4f8785f3473c939fc382769c90"
server_key_iv: "e428b8113968f5e21aedd1673b23e8c0"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.441967 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:55.468719 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:55.471243 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:55.471426 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:55.471652 12963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:55.472002 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:55.472044 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.472074 12963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:55.472100 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.487357 12963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33099
I20260924 00:11:55.487802 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:11:55.489284 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33099 every 8 connection(s)
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:55.498355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 12963
I20260924 00:11:55.498476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:11:55.498667 392 external_mini_cluster.cc:1428] Setting key 75ecc165adafd96d16b9b5e9a85cb6ba
I20260924 00:11:55.500573 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.490313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39525 (local address 127.0.98.62:39863)
0924 00:11:55.490359 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:55.490361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:55.490759 (+ 398us) server_negotiation.cc:407] Connection header received
0924 00:11:55.491463 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:55.491465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:55.491477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:55.491508 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:55.492309 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.493058 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.494506 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.494780 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.494819 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.497341 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:55.497343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:55.497343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:55.497351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:55.499015 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.499557 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.499558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.499559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.499616 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.499865 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.499866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.499866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.499992 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:55.500041 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:55.500389 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 00:11:55.500482 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:11:55.501077 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.490052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:39525)
0924 00:11:55.490634 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:55.490678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:55.491328 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:55.491690 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:55.491694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:55.491936 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:55.492206 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.492211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.493172 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.493173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.494360 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.494363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.494904 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.494905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.495038 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.495678 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:55.495689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:55.497235 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:55.499113 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.499119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.499122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.499471 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.499682 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.499682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.499683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.499799 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.500072 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:55.500077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:55.500247 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 00:11:55.500696 (+ 449us) client_negotiation.cc:240] Negotiation successful
0924 00:11:55.500853 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":74,"threads_started":1}
I20260924 00:11:55.501478 13092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:55.501560 13092 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.501730 13092 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:55.502044 12608 ts_manager.cc:194] Registered new tserver with Master: 73a09d99f1c94fb4bf368cf60354d561 (127.0.98.3:33099)
I20260924 00:11:55.502492 12608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39525
I20260924 00:11:55.510317 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:11:55.511003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 12689
I20260924 00:11:55.518322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 12826
I20260924 00:11:55.525811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 12963
I20260924 00:11:55.530932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 12576
I20260924 00:11:55.536455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--webserver_interface=127.0.98.62
--webserver_port=43437
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:55.618608 13095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:55.618856 13095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:55.618896 13095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:55.620096 13095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:11:55.620165 13095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:55.620191 13095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:55.620224 13095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:11:55.620247 13095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:11:55.621860 13095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863
--ranger_config_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:55.622131 13095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:55.622357 13095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:55.628098 13101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.628193 13103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.628295 13095 server_base.cc:1034] running on GCE node
W20260924 00:11:55.628093 13100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.628561 13095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:55.628881 13095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:55.630038 13095 hybrid_clock.cc:648] HybridClock initialized: now 1790208715630024 us; error 27 us; skew 500 ppm
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:55.632797 13095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:11:55.633328 13095 webserver.cc:460] Webserver started at http://127.0.98.62:43437/ using document root <none> and password file <none>
I20260924 00:11:55.633589 13095 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:55.633656 13095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:55.684026 13095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:11:55.684803 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.685026 13095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:11:55.685115 13095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "17552fe783b444d3ad7de71f0e613a94"
format_stamp: "Formatted at 2026-09-24 00:11:53 on dist-test-slave-3q8k"
server_key: "b58e6bdf8e55a358a5ab2b722c055c57"
server_key_iv: "73cc39a00f42e4ecddabda2824c65513"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.685518 13095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:55.706897 13095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:55.708917 13095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:55.709079 13095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:55.714573 13095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39863
I20260924 00:11:55.714614 13163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39863 every 8 connection(s)
I20260924 00:11:55.715032 13095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:11:55.716418 13164 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:11:55.717159 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Bootstrap starting.
I20260924 00:11:55.722323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13095
I20260924 00:11:55.722627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36091
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=45537
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:11:55.727000 13164 log.cc:824] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:55.729619 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:11:55.730005 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Bootstrap complete.
I20260924 00:11:55.734756 13164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:55.735198 13164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17552fe783b444d3ad7de71f0e613a94, State: Initialized, Role: FOLLOWER
I20260924 00:11:55.735471 13164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:55.735576 13164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:11:55.735630 13164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:11:55.735672 13164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:11:55.737255 13164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:55.737371 13164 leader_election.cc:302] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17552fe783b444d3ad7de71f0e613a94; no voters:
I20260924 00:11:55.737537 13164 leader_election.cc:288] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:11:55.737670 13168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:11:55.737809 13164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:11:55.737924 13168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [term 2 LEADER]: Becoming Leader. State: Replica: 17552fe783b444d3ad7de71f0e613a94, State: Running, Role: LEADER
I20260924 00:11:55.738065 13168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260924 00:11:55.738411 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "17552fe783b444d3ad7de71f0e613a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 00:11:55.738546 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:55.738718 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17552fe783b444d3ad7de71f0e613a94. Latest consensus state: current_term: 2 leader_uuid: "17552fe783b444d3ad7de71f0e613a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17552fe783b444d3ad7de71f0e613a94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260924 00:11:55.738816 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94 [sys.catalog]: This master's current role is: LEADER
I20260924 00:11:55.738950 13177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:11:55.739513 13177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:11:55.739691 13177 catalog_manager.cc:1259] Loaded cluster ID: a5aef91b4ce54127ba4d0c0d72a61c1f
I20260924 00:11:55.739769 13177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:11:55.739872 13164 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:11:55.741094 13177 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:11:55.741721 13177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Loaded TSK: 0
I20260924 00:11:55.741995 13177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:11:55.844622 13166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:55.844901 13166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:55.844937 13166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:55.846551 13166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:55.846609 13166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:55.846648 13166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:11:55.848851 13166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45537
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:55.849239 13166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:55.849521 13166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:55.856336 13204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.856467 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:55.856355 13202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:55.857242 13166 server_base.cc:1034] running on GCE node
I20260924 00:11:55.857467 13166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:55.857738 13166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:55.858913 13166 hybrid_clock.cc:648] HybridClock initialized: now 1790208715858891 us; error 36 us; skew 500 ppm
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:55.862551 13166 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:11:55.863170 13166 webserver.cc:460] Webserver started at http://127.0.98.1:45537/ using document root <none> and password file <none>
I20260924 00:11:55.863551 13166 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:55.863660 13166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:55.891731 13166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260924 00:11:55.895370 13212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.895624 13166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 00:11:55.895716 13166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "df60bbcc5e7e440abbf1872f21041b68"
format_stamp: "Formatted at 2026-09-24 00:11:54 on dist-test-slave-3q8k"
server_key: "698498e51d4e0ac2a63695127ef48536"
server_key_iv: "a711a3512c0db8b3e5d0057fe2aef5f5"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:55.896198 13166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:55.911788 13166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:55.914409 13166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:55.914623 13166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:55.914920 13166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:55.915375 13166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:55.915458 13166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.915518 13166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:55.915553 13166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:55.921794 13166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36091
I20260924 00:11:55.922307 13166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:11:55.924099 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36091 every 8 connection(s)
I20260924 00:11:55.932297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13166
I20260924 00:11:55.932636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33707
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=44119
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:11:55 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:55.942982 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.927939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:48491)
0924 00:11:55.928231 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:55.928283 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:55.928913 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:55.931860 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:55.931865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:55.932121 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:55.932460 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.932467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.934493 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.934494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.935762 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:55.935765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.936606 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.936607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:55.936754 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.937395 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:55.937405 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:55.938948 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:55.941052 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.941059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.941062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.941388 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.941687 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:55.941688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:55.941689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:55.941814 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:55.942268 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:55.942273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:55.942450 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 00:11:55.942535 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:11:55.942697 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260924 00:11:55.943607 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:55.927987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48491 (local address 127.0.98.62:39863)
0924 00:11:55.931383 (+ 3396us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:55.931393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:55.931433 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:11:55.931584 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:55.931591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:55.931644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:55.932586 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:55.932682 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.934373 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.935880 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:55.936483 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:55.937034 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:55.939061 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:55.939065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:55.939065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:55.939076 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:55.940948 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.941497 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.941500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.941501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.941610 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:55.941889 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:55.941890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:55.941890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:55.942186 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:55.943156 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:55.943261 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:11:55.943393 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3335,"thread_start_us":72,"threads_started":1}
I20260924 00:11:55.944351 13326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:55.944449 13326 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.944657 13326 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:55.945542 13326 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:11:55.945616 13326 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.945688 13326 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:55.946118 13326 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:55.946197 13326 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:55.946579 13326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:11:56.045023 13330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:56.045279 13330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:56.045316 13330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:56.046988 13330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:56.047047 13330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:56.047086 13330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:11:56.049252 13330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44119
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:56.049507 13330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:56.049815 13330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:56.057066 13338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:56.056927 13336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:56.056715 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:56.059619 13330 server_base.cc:1034] running on GCE node
I20260924 00:11:56.059885 13330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:56.060173 13330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:56.061367 13330 hybrid_clock.cc:648] HybridClock initialized: now 1790208716061353 us; error 28 us; skew 500 ppm
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:56.064651 13330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:11:56.065223 13330 webserver.cc:460] Webserver started at http://127.0.98.2:44119/ using document root <none> and password file <none>
I20260924 00:11:56.065521 13330 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:56.065608 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:56.080672 13330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:11:56.081430 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.081593 13330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:11:56.081681 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "20ce4e3edbb2446c9904ad4750f1b5a0"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "ef36c905f0ec5ee32875cdff59f07a01"
server_key_iv: "468b5cd1b287744a3d0ac10a62bd77d8"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:56.082070 13330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:56.103096 13330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:56.105530 13330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:56.105670 13330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:56.105916 13330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:56.106292 13330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:56.106338 13330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.106369 13330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:56.106393 13330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.113667 13330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33707
I20260924 00:11:56.113828 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33707 every 8 connection(s)
I20260924 00:11:56.114041 13330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:11:56.117811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13330
I20260924 00:11:56.118080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33099
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=41639
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:56.125108 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:56.114869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:46513)
0924 00:11:56.115041 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:56.115077 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:56.115620 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:56.115887 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:56.115890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:56.116072 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:56.116257 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:56.116261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.117184 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.117186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:56.118317 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:56.118320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.118901 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.118902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:56.119040 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:56.119654 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:56.119664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:56.121148 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:56.123195 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:56.123202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:56.123204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:56.123584 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:56.123827 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:56.123827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:56.123828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:56.123973 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:56.124326 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:56.124332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:56.124526 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 00:11:56.124643 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 00:11:56.124815 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260924 00:11:56.125531 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:56.114893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46513 (local address 127.0.98.62:39863)
0924 00:11:56.114957 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:56.114959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:56.115188 (+ 229us) server_negotiation.cc:407] Connection header received
0924 00:11:56.115738 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:56.115740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:56.115756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:56.115806 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:56.116360 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.117070 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.118420 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.118742 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.118784 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:56.121254 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:56.121256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:56.121257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:56.121265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:56.123093 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.123684 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.123686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.123686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.123750 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.124080 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.124082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.124082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.124230 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:56.124311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:56.125283 (+ 972us) server_negotiation.cc:299] Negotiation successful
0924 00:11:56.125393 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:11:56.125955 13460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:56.126067 13460 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.126277 13460 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:56.127054 13460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:11:56.127134 13460 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.127234 13460 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:56.127676 13460 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.127764 13460 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:56.128186 13460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:11:56.228291 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:56.228564 13463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:56.228598 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:56.230290 13463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:56.230348 13463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:56.230389 13463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:11:56.232672 13463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41639
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:56.232933 13463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:56.233196 13463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:56.239602 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:56.239713 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:56.239616 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:56.240252 13463 server_base.cc:1034] running on GCE node
I20260924 00:11:56.240473 13463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:56.240770 13463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:56.242009 13463 hybrid_clock.cc:648] HybridClock initialized: now 1790208716241994 us; error 28 us; skew 500 ppm
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:56.246168 13463 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:11:56.246706 13463 webserver.cc:460] Webserver started at http://127.0.98.3:41639/ using document root <none> and password file <none>
I20260924 00:11:56.246986 13463 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:56.247062 13463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:56.268590 13463 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:11:56.271373 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.271646 13463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 00:11:56.271750 13463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "73a09d99f1c94fb4bf368cf60354d561"
format_stamp: "Formatted at 2026-09-24 00:11:55 on dist-test-slave-3q8k"
server_key: "5fc6eb4f8785f3473c939fc382769c90"
server_key_iv: "e428b8113968f5e21aedd1673b23e8c0"
server_key_version: "kuduclusterkey@0"
I20260924 00:11:56.272156 13463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:56.289726 13463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:56.292176 13463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:56.292341 13463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:56.292582 13463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:11:56.292984 13463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:11:56.293026 13463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.293049 13463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:11:56.293061 13463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:56.300238 13463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33099
I20260924 00:11:56.300712 13463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:11:56.301570 13593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33099 every 8 connection(s)
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:56.308105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13463
I20260924 00:11:56.312338 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:56.301910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34873 (local address 127.0.98.62:39863)
0924 00:11:56.301956 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:56.301958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:56.301971 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:11:56.302514 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:56.302516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:56.302528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:56.302604 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:56.303261 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.304048 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.305157 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.305468 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.305543 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:56.308855 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:56.308857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:56.308858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:56.308866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:56.310625 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.311150 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.311152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.311152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.311242 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.311568 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.311569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.311569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.311701 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:56.311747 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:56.312151 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 00:11:56.312239 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:11:56.313098 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:56.301586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:34873)
0924 00:11:56.301829 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:56.301877 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:56.302389 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:56.302659 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:56.302662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:56.302850 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:56.303142 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:56.303148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.304146 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.304146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:56.305016 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:56.305019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.305624 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.305625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:56.305773 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:56.306509 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:56.306519 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:56.308741 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:56.310722 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:56.310729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:56.310731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:56.311053 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:56.311324 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:56.311325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:56.311325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:56.311470 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:56.311797 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:56.311803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:56.311991 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 00:11:56.312689 (+ 698us) client_negotiation.cc:240] Negotiation successful
0924 00:11:56.312869 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260924 00:11:56.314137 13594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260924 00:11:56.314275 13594 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.314507 13594 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:56.315142 13594 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:11:56.315284 13594 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.315404 13594 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:56.315744 13594 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.315863 13594 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
W20260924 00:11:56.316179 13594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:11:56.318478 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:56.308638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.0.98.62:39863)
0924 00:11:56.310613 (+ 1975us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:56.310616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:56.310622 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:11:56.310643 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:56.310644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:56.310656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:56.310718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:56.311541 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.312319 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.313429 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:56.313764 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:56.313868 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:56.315420 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:56.315422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:56.315422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:56.315429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:56.317288 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.317724 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.317725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.317725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.317793 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:56.318035 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:56.318035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:56.318036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:56.318138 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:56.318252 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:56.318346 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 00:11:56.318410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1947,"thread_start_us":50,"threads_started":1}
Sep 24 00:11:56 dist-test-slave-3q8k krb5kdc[12132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:56.946790 13326 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:56.946990 13326 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:56.947499 13128 ts_manager.cc:194] Registered new tserver with Master: df60bbcc5e7e440abbf1872f21041b68 (127.0.98.1:36091)
I20260924 00:11:56.948494 13128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48491
I20260924 00:11:57.128381 13460 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:57.128542 13460 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:57.128847 13128 ts_manager.cc:194] Registered new tserver with Master: 20ce4e3edbb2446c9904ad4750f1b5a0 (127.0.98.2:33707)
I20260924 00:11:57.129401 13128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46513
Sep 24 00:11:57 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:11:57.316391 13594 heartbeater.cc:464] Registering TS with master...
I20260924 00:11:57.316545 13594 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260924 00:11:57.316876 13128 ts_manager.cc:194] Registered new tserver with Master: 73a09d99f1c94fb4bf368cf60354d561 (127.0.98.3:33099)
I20260924 00:11:57.317395 13128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34873
I20260924 00:11:57.321300 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:11:57.330267 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.322252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.98.62:39863)
0924 00:11:57.322554 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.322557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.322564 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:11:57.322585 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.322586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.322597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.322675 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:57.323393 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.324178 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.325219 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.325524 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.325605 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.327092 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:57.327093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:57.327093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:57.327101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:57.328927 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:57.329313 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:57.329314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:57.329315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:57.329382 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:57.329595 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:57.329596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:57.329596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:57.329734 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:57.329858 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.329972 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.330060 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":53,"threads_started":1}
I20260924 00:11:57.332459 13128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:11:57.334877 13128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:11:57.343597 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.338056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.98.2:33707)
0924 00:11:57.338270 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.338276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.338847 (+ 571us) server_negotiation.cc:407] Connection header received
0924 00:11:57.338969 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.338971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.338988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.339040 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:57.339381 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.340686 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.342303 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.343050 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.343099 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.343175 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.343473 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.343514 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":54,"threads_started":1}
I20260924 00:11:57.343954 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.338105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.0.98.1:36091)
0924 00:11:57.338327 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.338330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.338752 (+ 422us) server_negotiation.cc:407] Connection header received
0924 00:11:57.338909 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.338912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.338928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.338980 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:57.339359 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.340517 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.342365 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.343168 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.343242 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.343320 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.343824 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.343870 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":66,"threads_started":1}
I20260924 00:11:57.344069 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.337799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33707 (local address 127.0.0.1:60068)
0924 00:11:57.338786 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.338796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.338886 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.339146 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:57.339149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:57.339151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:57.339269 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.339272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.340771 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.340771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.342200 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.342203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.343271 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.343272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.343353 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.343367 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:11:57.343927 (+ 560us) client_negotiation.cc:240] Negotiation successful
0924 00:11:57.343992 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":97,"thread_start_us":92,"threads_started":1}
I20260924 00:11:57.344445 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.337788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36091 (local address 127.0.0.1:39258)
0924 00:11:57.338644 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.338686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.338806 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.339146 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:57.339149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:57.339156 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:57.339265 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.339268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.340805 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.340806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.342256 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.342259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.343652 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.343653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.343720 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.343723 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:11:57.344320 (+ 597us) client_negotiation.cc:240] Negotiation successful
0924 00:11:57.344368 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260924 00:11:57.344841 13394 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd (DEFAULT_TABLE table=test-table [id=92ca8d0ad7e54718b28fba3744ce2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:57.345204 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:57.345121 13258 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd (DEFAULT_TABLE table=test-table [id=92ca8d0ad7e54718b28fba3744ce2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:57.345386 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:57.348887 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.338238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.0.1:48586)
0924 00:11:57.344377 (+ 6139us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.344388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.344427 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.345030 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:57.345031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:57.345033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:57.345061 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.345063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.346570 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.346571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.347645 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.347648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.348657 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.348658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.348713 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.348716 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:11:57.348778 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:11:57.348815 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6128,"thread_start_us":60,"threads_started":1}
I20260924 00:11:57.349097 13629 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Bootstrap starting.
I20260924 00:11:57.349190 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.338418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.0.98.3:33099)
0924 00:11:57.344800 (+ 6382us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.344805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.344815 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:11:57.344853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.344855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.344872 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.344951 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:57.345130 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.346474 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.347744 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.348481 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.348551 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.348623 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.349071 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.349115 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6332,"thread_start_us":66,"threads_started":1}
I20260924 00:11:57.349804 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd (DEFAULT_TABLE table=test-table [id=92ca8d0ad7e54718b28fba3744ce2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:11:57.350136 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:57.351382 13628 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Bootstrap starting.
I20260924 00:11:57.352147 13628 tablet_bootstrap.cc:654] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:57.352533 13628 log.cc:824] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:57.354017 13629 tablet_bootstrap.cc:654] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:57.354310 13629 log.cc:824] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:57.354594 13631 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Bootstrap starting.
I20260924 00:11:57.355178 13631 tablet_bootstrap.cc:654] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Neither blocks nor log segments found. Creating new log.
I20260924 00:11:57.355473 13631 log.cc:824] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Log is configured to *not* fsync() on all Append() calls
I20260924 00:11:57.356216 13629 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: No bootstrap required, opened a new log
I20260924 00:11:57.356289 13629 ts_tablet_manager.cc:1397] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 00:11:57.359304 13629 raft_consensus.cc:348] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.359481 13629 raft_consensus.cc:374] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:57.359519 13629 raft_consensus.cc:729] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df60bbcc5e7e440abbf1872f21041b68, State: Initialized, Role: FOLLOWER
I20260924 00:11:57.359654 13629 consensus_queue.cc:260] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.359922 13629 ts_tablet_manager.cc:1428] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:11:57.360044 13326 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260924 00:11:57.360323 13628 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: No bootstrap required, opened a new log
I20260924 00:11:57.360453 13628 ts_tablet_manager.cc:1397] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 00:11:57.361752 13631 tablet_bootstrap.cc:492] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: No bootstrap required, opened a new log
I20260924 00:11:57.361825 13631 ts_tablet_manager.cc:1397] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 00:11:57.363930 13628 raft_consensus.cc:348] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.364179 13628 raft_consensus.cc:374] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:57.364276 13628 raft_consensus.cc:729] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ce4e3edbb2446c9904ad4750f1b5a0, State: Initialized, Role: FOLLOWER
I20260924 00:11:57.364475 13628 consensus_queue.cc:260] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.364451 13631 raft_consensus.cc:348] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.364619 13631 raft_consensus.cc:374] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:11:57.364661 13631 raft_consensus.cc:729] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a09d99f1c94fb4bf368cf60354d561, State: Initialized, Role: FOLLOWER
I20260924 00:11:57.364921 13628 ts_tablet_manager.cc:1428] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 00:11:57.365089 13460 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
W20260924 00:11:57.365331 13461 tablet.cc:2369] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:11:57.364800 13631 consensus_queue.cc:260] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.365760 13631 ts_tablet_manager.cc:1428] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:11:57.365880 13594 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260924 00:11:57.419270 13634 raft_consensus.cc:482] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:11:57.419520 13634 raft_consensus.cc:504] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.420862 13634 leader_election.cc:288] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ce4e3edbb2446c9904ad4750f1b5a0 (127.0.98.2:33707), 73a09d99f1c94fb4bf368cf60354d561 (127.0.98.3:33099)
I20260924 00:11:57.426103 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.420801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57531 (local address 127.0.98.3:33099)
0924 00:11:57.420842 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.420843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.421640 (+ 797us) server_negotiation.cc:407] Connection header received
0924 00:11:57.421713 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.421714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.421728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.421796 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:57.421984 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.422531 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.424394 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.424987 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.425038 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.425050 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.425973 (+ 923us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.426025 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:11:57.426381 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.420388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46645 (local address 127.0.98.2:33707)
0924 00:11:57.420436 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.420439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.421339 (+ 900us) server_negotiation.cc:407] Connection header received
0924 00:11:57.422057 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.422058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.422072 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.422124 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:11:57.422773 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.423522 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.425245 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.425844 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.425904 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.425915 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.426268 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.426311 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:11:57.426597 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.420174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33707 (local address 127.0.98.1:46645)
0924 00:11:57.421252 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.421263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.421328 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.422625 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:57.422626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:57.422628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:57.422693 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.422694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.423637 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.423639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.425136 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.425140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.426112 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.426112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.426189 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.426192 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:11:57.426469 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 00:11:57.426513 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":103,"threads_started":1}
I20260924 00:11:57.426872 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.420605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.98.1:57531)
0924 00:11:57.421556 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.421570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.421653 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.421814 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:57.421815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:57.421817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:11:57.421912 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.421914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.422733 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.422734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.424162 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:57.424166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.425791 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.425792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:57.425863 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.425867 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:11:57.426759 (+ 892us) client_negotiation.cc:240] Negotiation successful
0924 00:11:57.426796 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":79,"threads_started":1}
I20260924 00:11:57.426954 13401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ca0d6b71a4291b69a6d91bdec29fd" candidate_uuid: "df60bbcc5e7e440abbf1872f21041b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" is_pre_election: true
I20260924 00:11:57.427137 13401 raft_consensus.cc:2454] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df60bbcc5e7e440abbf1872f21041b68 in term 0.
I20260924 00:11:57.427179 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ca0d6b71a4291b69a6d91bdec29fd" candidate_uuid: "df60bbcc5e7e440abbf1872f21041b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a09d99f1c94fb4bf368cf60354d561" is_pre_election: true
I20260924 00:11:57.427356 13548 raft_consensus.cc:2454] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df60bbcc5e7e440abbf1872f21041b68 in term 0.
I20260924 00:11:57.427433 13215 leader_election.cc:302] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ce4e3edbb2446c9904ad4750f1b5a0, df60bbcc5e7e440abbf1872f21041b68; no voters:
I20260924 00:11:57.427572 13634 raft_consensus.cc:2781] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:11:57.427636 13634 raft_consensus.cc:482] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:11:57.427661 13634 raft_consensus.cc:3037] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:11:57 dist-test-slave-3q8k krb5kdc[12132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 00:11:57.428288 13327 tablet.cc:2369] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:11:57.428503 13634 raft_consensus.cc:504] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.428633 13634 leader_election.cc:288] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [CANDIDATE]: Term 1 election: Requested vote from peers 20ce4e3edbb2446c9904ad4750f1b5a0 (127.0.98.2:33707), 73a09d99f1c94fb4bf368cf60354d561 (127.0.98.3:33099)
I20260924 00:11:57.428781 13401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ca0d6b71a4291b69a6d91bdec29fd" candidate_uuid: "df60bbcc5e7e440abbf1872f21041b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0"
I20260924 00:11:57.428851 13401 raft_consensus.cc:3037] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:57.428911 13548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ca0d6b71a4291b69a6d91bdec29fd" candidate_uuid: "df60bbcc5e7e440abbf1872f21041b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a09d99f1c94fb4bf368cf60354d561"
I20260924 00:11:57.428978 13548 raft_consensus.cc:3037] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:11:57.429705 13401 raft_consensus.cc:2454] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df60bbcc5e7e440abbf1872f21041b68 in term 1.
I20260924 00:11:57.429733 13548 raft_consensus.cc:2454] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df60bbcc5e7e440abbf1872f21041b68 in term 1.
I20260924 00:11:57.429908 13213 leader_election.cc:302] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a09d99f1c94fb4bf368cf60354d561, df60bbcc5e7e440abbf1872f21041b68; no voters:
I20260924 00:11:57.430017 13634 raft_consensus.cc:2781] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:11:57.430142 13634 raft_consensus.cc:686] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [term 1 LEADER]: Becoming Leader. State: Replica: df60bbcc5e7e440abbf1872f21041b68, State: Running, Role: LEADER
I20260924 00:11:57.430222 13634 consensus_queue.cc:237] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } }
I20260924 00:11:57.431006 13127 catalog_manager.cc:5495] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 reported cstate change: term changed from 0 to 1, leader changed from <none> to df60bbcc5e7e440abbf1872f21041b68 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "df60bbcc5e7e440abbf1872f21041b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df60bbcc5e7e440abbf1872f21041b68" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } health_report { overall_health: UNKNOWN } } }
I20260924 00:11:57.460510 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:57.457307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.0.98.1:36091)
0924 00:11:57.457356 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:57.457358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:57.457369 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:11:57.457441 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:57.457442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:57.457465 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:57.457541 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:11:57.457696 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.458210 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.459341 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:57.459623 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:57.459692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:57.459902 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:11:57.460068 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:11:57.460163 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:57.460405 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 00:11:57.460428 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:11:57.462472 13548 raft_consensus.cc:1261] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 1 FOLLOWER]: Refusing update from remote peer df60bbcc5e7e440abbf1872f21041b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:11:57.462822 13634 consensus_queue.cc:1035] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a09d99f1c94fb4bf368cf60354d561" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:11:57.463040 13401 raft_consensus.cc:1261] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 1 FOLLOWER]: Refusing update from remote peer df60bbcc5e7e440abbf1872f21041b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:11:57.463315 13634 consensus_queue.cc:1035] T 1b4ca0d6b71a4291b69a6d91bdec29fd P df60bbcc5e7e440abbf1872f21041b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ce4e3edbb2446c9904ad4750f1b5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:11:57.466090 13643 mvcc.cc:205] Tried to move back new op lower bound from 7332694906723143680 to 7332694906594684928. Current Snapshot: MvccSnapshot[applied={T|T < 7332694906723143680}]
I20260924 00:11:57.467664 13646 mvcc.cc:205] Tried to move back new op lower bound from 7332694906723143680 to 7332694906594684928. Current Snapshot: MvccSnapshot[applied={T|T < 7332694906723143680}]
I20260924 00:11:57.468432 13128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:11:57.468523 13128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:11:57.469254 13128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17552fe783b444d3ad7de71f0e613a94: Sending DeleteTablet for 3 replicas of tablet 1b4ca0d6b71a4291b69a6d91bdec29fd
I20260924 00:11:57.469587 13258 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39258
I20260924 00:11:57.470098 13394 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60068
I20260924 00:11:57.470364 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:11:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48586
I20260924 00:11:57.471117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13166
I20260924 00:11:57.475637 13644 mvcc.cc:205] Tried to move back new op lower bound from 7332694906723143680 to 7332694906594684928. Current Snapshot: MvccSnapshot[applied={T|T < 7332694906723143680}]
I20260924 00:11:57.475823 13655 tablet_replica.cc:330] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: stopping tablet replica
I20260924 00:11:57.476442 13654 tablet_replica.cc:330] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0: stopping tablet replica
I20260924 00:11:57.476577 13654 raft_consensus.cc:2229] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:11:57.476667 13654 raft_consensus.cc:2258] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 20ce4e3edbb2446c9904ad4750f1b5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:11:57.478935 13655 raft_consensus.cc:2229] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:11:57.479034 13655 raft_consensus.cc:2258] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:11:57.480727 13655 ts_tablet_manager.cc:1905] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:11:57.482744 13655 ts_tablet_manager.cc:1918] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:11:57.482826 13655 log.cc:1196] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/1b4ca0d6b71a4291b69a6d91bdec29fd
I20260924 00:11:57.483091 13655 ts_tablet_manager.cc:1939] T 1b4ca0d6b71a4291b69a6d91bdec29fd P 73a09d99f1c94fb4bf368cf60354d561: Deleting consensus metadata
I20260924 00:11:57.483609 13112 catalog_manager.cc:4841] TS 73a09d99f1c94fb4bf368cf60354d561 (127.0.98.3:33099): tablet 1b4ca0d6b71a4291b69a6d91bdec29fd (table test-table [id=92ca8d0ad7e54718b28fba3744ce2e78]) successfully deleted
W20260924 00:11:57.483994 13113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36091 (error 108)
W20260924 00:11:57.485803 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:11:57.485494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36091 (local address 127.0.0.1:39276)
0924 00:11:57.485562 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.485577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.485633 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.485756 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 00:11:57.486076 13113 catalog_manager.cc:4584] TS df60bbcc5e7e440abbf1872f21041b68 (127.0.98.1:36091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b4ca0d6b71a4291b69a6d91bdec29fd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:11:57.486532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13330
W20260924 00:11:57.497536 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:11:57.496748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33707 (local address 127.0.0.1:60070)
0924 00:11:57.496803 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:57.496814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:57.496855 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:57.497474 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:11:57.497764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13463
I20260924 00:11:57.503711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13095
I20260924 00:11:57.515720 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 00:11:57.860580 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 00:11:57.860711 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:11:58.205097 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:11:58.205 UTC [12162] LOG: received smart shutdown request
2026-09-24 00:11:58.208 UTC [12162] LOG: background worker "logical replication launcher" (PID 12171) exited with exit code 1
2026-09-24 00:11:58.209 UTC [12166] LOG: shutting down
2026-09-24 00:11:58.388 UTC [12162] LOG: database system is shut down
2026-09-24T00:11:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48080 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:11:58 dist-test-slave-3q8k krb5kdc[13666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:11:58 dist-test-slave-3q8k krb5kdc[13666](info): set up 2 sockets
Sep 24 00:11:58 dist-test-slave-3q8k krb5kdc[13666](info): commencing operation
krb5kdc: starting...
W20260924 00:11:59.539274 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:11:59.657663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:11:59.742118 13679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:59.742426 13679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:59.742458 13679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:59.744010 13679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:11:59.744071 13679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:59.744097 13679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:59.744127 13679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:11:59.744153 13679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:11:59.746001 13679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:59.746294 13679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:59.746531 13679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:59.751391 13687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:59.751403 13684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:59.751395 13685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:59.751441 13679 server_base.cc:1034] running on GCE node
I20260924 00:11:59.751960 13679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:59.752261 13679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:59.753400 13679 hybrid_clock.cc:648] HybridClock initialized: now 1790208719753379 us; error 40 us; skew 500 ppm
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:59.760599 13679 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:11:59.761051 13679 webserver.cc:460] Webserver started at http://127.0.98.62:41877/ using document root <none> and password file <none>
I20260924 00:11:59.761242 13679 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:59.761283 13679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:59.761389 13679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:11:59.762432 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "50cc40ece84049619100c32fe22e40ca"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "e4d790ebcf8378d00c3a2f5471158f69"
server_key_iv: "cc2e107339a917d8521ad6ca4ad506b8"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.762806 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "50cc40ece84049619100c32fe22e40ca"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "e4d790ebcf8378d00c3a2f5471158f69"
server_key_iv: "cc2e107339a917d8521ad6ca4ad506b8"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.764125 13679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:11:59.764796 13694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:59.764954 13679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:59.765056 13679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "50cc40ece84049619100c32fe22e40ca"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "e4d790ebcf8378d00c3a2f5471158f69"
server_key_iv: "cc2e107339a917d8521ad6ca4ad506b8"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.765126 13679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:59.776602 13679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:59.778388 13679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:59.778556 13679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:59.782671 13679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42461
I20260924 00:11:59.782706 13746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42461 every 8 connection(s)
I20260924 00:11:59.783068 13679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:11:59.783919 13747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:59.785480 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:59.793061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13679
I20260924 00:11:59.793188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:11:59.793401 392 external_mini_cluster.cc:1428] Setting key cefdbac1e5a952fa2610057e5b3fa543
I20260924 00:11:59.794889 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.786184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.0.98.62:42461)
0924 00:11:59.786338 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:59.786345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:59.786416 (+ 71us) server_negotiation.cc:407] Connection header received
0924 00:11:59.786929 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:59.786931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:59.786946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:59.787029 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:59.787615 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.788399 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.789233 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.789551 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.789626 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.791782 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:59.791789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:59.791790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:59.791796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:59.793333 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.793713 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.793716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.793717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.793819 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.794038 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.794039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.794039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.794340 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:59.794421 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:59.794588 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 00:11:59.794745 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 00:11:59.794870 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.786143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.0.1:58422)
0924 00:11:59.786303 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.786342 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:59.786808 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:59.787053 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:59.787055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:59.787379 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:59.787536 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.787540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.788486 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.788487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.789164 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.789166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.789629 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.789629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.789686 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.790277 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:59.790287 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:59.791707 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:59.793410 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.793416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.793418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.793638 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.793898 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.793899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.793899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.793978 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.794412 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:59.794416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:59.794470 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 00:11:59.794536 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:11:59.794659 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260924 00:11:59.795681 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:11:59.796036 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.795902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56174)
0924 00:11:59.795949 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.796007 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 00:11:59.796178 13698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
I20260924 00:11:59.796595 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.796500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56182)
0924 00:11:59.796536 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.796564 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 00:11:59.796710 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41163: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:11:59.818137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41163
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:11:59.845973 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } attempt: 1
I20260924 00:11:59.846482 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.846312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56186)
0924 00:11:59.846387 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.846442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:11:59.846844 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.846734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56188)
0924 00:11:59.846791 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.846821 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 00:11:59.846956 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41163: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
W20260924 00:11:59.899717 13754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:11:59.899962 13754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:11:59.900002 13754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:11:59.901129 13754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:11:59.901191 13754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:11:59.901216 13754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:11:59.901255 13754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:11:59.901284 13754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:11:59.902773 13754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:11:59.903039 13754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:11:59.903301 13754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:11:59.908044 13760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:59.908058 13759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:11:59.908083 13762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:11:59.908316 13754 server_base.cc:1034] running on GCE node
I20260924 00:11:59.908607 13754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:11:59.908846 13754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:11:59.910006 13754 hybrid_clock.cc:648] HybridClock initialized: now 1790208719910000 us; error 36 us; skew 500 ppm
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:11:59.912582 13754 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:11:59.913007 13754 webserver.cc:460] Webserver started at http://127.0.98.61:45545/ using document root <none> and password file <none>
I20260924 00:11:59.913255 13754 fs_manager.cc:359] Metadata directory not provided
I20260924 00:11:59.913318 13754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:11:59.913465 13754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:11:59.914497 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data/instance:
uuid: "5bead804e7b94fc9b68f7eb34b678871"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "15cbbe30ccf2515b0483da0a4f21dc8f"
server_key_iv: "45d42fdf569e240e0d7892bdf5876531"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.914857 13754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal/instance:
uuid: "5bead804e7b94fc9b68f7eb34b678871"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "15cbbe30ccf2515b0483da0a4f21dc8f"
server_key_iv: "45d42fdf569e240e0d7892bdf5876531"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.915112 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } attempt: 2
I20260924 00:11:59.915513 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.915370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56192)
0924 00:11:59.915432 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.915477 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:11:59.915908 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.915817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56202)
0924 00:11:59.915861 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.915888 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:11:59.916038 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41163: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41163: connect: Connection refused (error 111)
I20260924 00:11:59.916528 13754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:11:59.917186 13769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:11:59.917356 13754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:11:59.917443 13754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "5bead804e7b94fc9b68f7eb34b678871"
format_stamp: "Formatted at 2026-09-24 00:11:59 on dist-test-slave-3q8k"
server_key: "15cbbe30ccf2515b0483da0a4f21dc8f"
server_key_iv: "45d42fdf569e240e0d7892bdf5876531"
server_key_version: "encryptionkey@0"
I20260924 00:11:59.917523 13754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:11:59.940533 13754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:11:59.942433 13754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:11:59.942593 13754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:11:59.946473 13754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41163
I20260924 00:11:59.946522 13821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41163 every 8 connection(s)
I20260924 00:11:59.946879 13754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:11:59.947772 13822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:11:59.949293 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:11:59.953157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13754
I20260924 00:11:59.953280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-1/wal/instance
I20260924 00:11:59.953506 392 external_mini_cluster.cc:1428] Setting key 3fe1941ae6d87b712ea9f020650bf6a5
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:11:59.958824 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.949985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.0.98.62:42461)
0924 00:11:59.950055 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:59.950058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:59.950272 (+ 214us) server_negotiation.cc:407] Connection header received
0924 00:11:59.950702 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:59.950703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:59.950714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:59.950768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:59.951407 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.951894 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.952821 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.953013 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.953065 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.955773 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:59.955775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:59.955775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:59.955782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:59.957171 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.957675 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.957676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.957676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.957738 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.958084 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.958085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.958085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.958206 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:59.958273 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:59.958663 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 00:11:59.958746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:11:59.959106 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.949977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.0.1:58426)
0924 00:11:59.950155 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.950190 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:59.950620 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:59.950841 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:59.950844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:59.951089 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:59.951337 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.951341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.951971 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.951971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.952748 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.952750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.953097 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.953098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.953258 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.953814 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:59.953824 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:59.955675 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:59.957251 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.957261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.957263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.957579 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.957839 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.957840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.957840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.958009 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.958287 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:59.958294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:59.958478 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 00:11:59.958613 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 00:11:59.958807 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260924 00:11:59.959703 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 00:11:59 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:11:59.967837 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.959991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56208)
0924 00:11:59.960063 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.960072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:11:59.960108 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:11:59.960390 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:11:59.960391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:11:59.960543 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:11:59.960590 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.960591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.961685 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.961686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.962249 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:11:59.962250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.962824 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.962825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:11:59.962896 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.963331 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:11:59.963335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:11:59.964748 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:11:59.966584 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.966585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.966586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.966821 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.967023 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:11:59.967024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:11:59.967024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:11:59.967096 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:11:59.967403 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:11:59.967404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:11:59.967453 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 00:11:59.967528 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:11:59.967641 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 00:11:59.967864 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.960059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.98.61:41163)
0924 00:11:59.960244 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:11:59.960248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:11:59.960266 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:11:59.960288 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:11:59.960290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:11:59.960302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:11:59.960390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:11:59.960667 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.961609 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.962357 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:11:59.962715 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:11:59.962817 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:11:59.964835 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:11:59.964845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:11:59.964846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:11:59.964853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:11:59.966462 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.966904 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.966906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.966906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.966958 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:11:59.967156 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:11:59.967156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:11:59.967156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:11:59.967340 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:11:59.967405 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:11:59.967563 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:11:59.967711 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":161,"thread_start_us":115,"threads_started":1}
I20260924 00:11:59.968729 13822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:11:59.969090 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.968963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43770)
0924 00:11:59.968999 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.969055 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 00:11:59.969229 13773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
I20260924 00:11:59.969646 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:11:59.969536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43786)
0924 00:11:59.969583 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:11:59.969616 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 00:11:59.969791 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:11:59.982148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34457
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:00.007273 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } attempt: 3
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:12:00.015336 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.007676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56224)
0924 00:12:00.007753 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.007768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.007841 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.008071 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.008072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.008451 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.008525 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.008527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.009490 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.009491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.009990 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.009991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.010511 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.010512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.010576 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.011069 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.011074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.012552 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.014244 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.014244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.014245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.014440 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.014667 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.014668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.014669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.014732 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.015017 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.015017 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.015056 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.015106 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.015189 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:12:00.015334 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.007743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.98.61:41163)
0924 00:12:00.007830 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.007832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.007846 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:12:00.007913 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.007914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.007929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.007993 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.008609 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.009394 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.010070 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.010417 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.010495 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.012632 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.012634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.012635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.012643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.014161 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.014513 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.014515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.014515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.014579 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.014796 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.014797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.014797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.014935 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.015027 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.015129 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.015206 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:12:00.016126 13747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:12:00.016479 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.016363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43798)
0924 00:12:00.016408 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.016435 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 00:12:00.016793 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.016714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43814)
0924 00:12:00.016756 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.016778 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 00:12:00.016896 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
I20260924 00:12:00.019026 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 1
I20260924 00:12:00.019384 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.019283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43816)
0924 00:12:00.019325 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.019358 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:12:00.019696 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.019612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43828)
0924 00:12:00.019653 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.019679 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:12:00.019835 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
I20260924 00:12:00.048080 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 1
I20260924 00:12:00.048664 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.048516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43842)
0924 00:12:00.048590 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.048630 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 00:12:00.048990 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.048900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43848)
0924 00:12:00.048941 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.048968 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 00:12:00.049081 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
W20260924 00:12:00.067056 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:00.067278 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:00.067301 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:00.068464 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:00.068513 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:00.068527 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:00.068571 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:00.068595 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:00.070108 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42461,127.0.98.61:41163,127.0.98.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:00.070358 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:00.070593 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:00.075281 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:00.075335 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:00.075433 13829 server_base.cc:1034] running on GCE node
W20260924 00:12:00.075281 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:00.075800 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:00.076107 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:00.077291 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790208720077273 us; error 37 us; skew 500 ppm
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:00.080325 13829 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:12:00.080746 13829 webserver.cc:460] Webserver started at http://127.0.98.60:36061/ using document root <none> and password file <none>
I20260924 00:12:00.080976 13829 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:00.081039 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:00.081188 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:00.082242 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data/instance:
uuid: "de62f084bc564928b0895f5aa242a8e6"
format_stamp: "Formatted at 2026-09-24 00:12:00 on dist-test-slave-3q8k"
server_key: "bb642a2385b481ed727e67589d88ea76"
server_key_iv: "0c43b52084dc3da33208e987b930cba4"
server_key_version: "encryptionkey@0"
I20260924 00:12:00.082607 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal/instance:
uuid: "de62f084bc564928b0895f5aa242a8e6"
format_stamp: "Formatted at 2026-09-24 00:12:00 on dist-test-slave-3q8k"
server_key: "bb642a2385b481ed727e67589d88ea76"
server_key_iv: "0c43b52084dc3da33208e987b930cba4"
server_key_version: "encryptionkey@0"
I20260924 00:12:00.083971 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:12:00.084664 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:00.084832 13829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:00.084928 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "de62f084bc564928b0895f5aa242a8e6"
format_stamp: "Formatted at 2026-09-24 00:12:00 on dist-test-slave-3q8k"
server_key: "bb642a2385b481ed727e67589d88ea76"
server_key_iv: "0c43b52084dc3da33208e987b930cba4"
server_key_version: "encryptionkey@0"
I20260924 00:12:00.085008 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:00.087958 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 2
I20260924 00:12:00.088379 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.088260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43852)
0924 00:12:00.088317 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.088352 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 00:12:00.088691 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.088608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43864)
0924 00:12:00.088648 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.088671 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 00:12:00.088773 13822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
I20260924 00:12:00.121042 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:00.122936 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:00.123109 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:00.124231 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 2
I20260924 00:12:00.124661 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.124500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43866)
0924 00:12:00.124574 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.124617 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:12:00.125074 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.124937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43876)
0924 00:12:00.125009 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.125042 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 00:12:00.125180 13747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34457: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34457: connect: Connection refused (error 111)
I20260924 00:12:00.127290 13829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34457
I20260924 00:12:00.127328 13896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34457 every 8 connection(s)
I20260924 00:12:00.127684 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:12:00.128515 13897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:00.130085 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:00.137167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13829
I20260924 00:12:00.137284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790208568194258-392-0/minicluster-data/master-2/wal/instance
I20260924 00:12:00.137480 392 external_mini_cluster.cc:1428] Setting key 914e0009af9eabc758544d72b7a2c05c
I20260924 00:12:00.138828 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.130784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.98.62:42461)
0924 00:12:00.130839 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.130841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.131011 (+ 170us) server_negotiation.cc:407] Connection header received
0924 00:12:00.131502 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.131503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.131511 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.131568 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.132164 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.132623 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.133549 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.133728 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.133784 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.135846 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.135847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.135847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.135854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.137256 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.137720 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.137721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.137722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.137781 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.138107 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.138108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.138108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.138287 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.138360 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.138617 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.138709 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:12:00.138849 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.130730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42461 (local address 127.0.0.1:58440)
0924 00:12:00.130885 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.130927 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.131421 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.131636 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.131640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.131892 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.132096 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.132099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.132696 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.132696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.133477 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.133478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.133801 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.133802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.133926 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.134396 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.134404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.135757 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.137362 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.137369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.137371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.137643 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.137869 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.137870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.137871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.138005 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.138353 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.138358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.138489 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.138573 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.138692 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260924 00:12:00.139671 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:12:00.146622 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.137859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.60:34457)
0924 00:12:00.138090 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.138096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.138136 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:12:00.138210 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.138212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.138223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.138280 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.138986 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.140052 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.140887 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.141229 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.141283 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.143792 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.143799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.143800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.143806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.145285 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.145653 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.145656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.145656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.145739 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.145923 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.145923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.145924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.146082 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.146232 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.146310 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.146395 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":131,"threads_started":1}
I20260924 00:12:00.146690 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.139977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.98.61:41163)
0924 00:12:00.140044 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.140046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.140056 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:12:00.140080 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.140081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.140092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.140146 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.140437 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.140894 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.141672 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.141925 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.141977 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.144098 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.144100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.144100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.144108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.145556 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.145943 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.145944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.145945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.146013 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.146185 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.146186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.146186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.146317 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.146379 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.146544 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.146614 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:12:00.146703 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.139901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41163 (local address 127.0.0.1:56236)
0924 00:12:00.139948 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.139967 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.140014 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.140158 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.140159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.140310 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.140364 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.140366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.140978 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.140979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.141596 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.141599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.141999 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.142000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.142072 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.142379 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.142385 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.144000 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.145647 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.145648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.145648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.145863 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.146069 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.146070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.146070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.146126 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.146382 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.146384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.146416 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.146495 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.146578 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 00:12:00.147240 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:00.154181 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.147529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.98.60:34457)
0924 00:12:00.147562 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.147564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.147573 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:00.147638 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.147639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.147649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.147705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.147976 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.148757 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.149646 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.149885 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.149946 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.151707 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.151708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.151709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.151717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.153092 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.153468 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.153468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.153469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.153518 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.153706 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.153707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.153707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.153796 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.153843 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.154025 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.154091 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 00:12:00.154311 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.147495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43896)
0924 00:12:00.147533 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.147542 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.147578 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.147706 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.147707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.147868 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.147906 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.147908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.148836 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.148837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.149572 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.149574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.149959 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.149960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.149996 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.150339 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.150344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.151630 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.153168 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.153169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.153169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.153375 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.153581 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.153582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.153583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.153643 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.153856 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.153857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.153904 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.154011 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.154151 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":48}
I20260924 00:12:00.155200 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.147493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.0.98.62:42461)
0924 00:12:00.147578 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.147580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.147588 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:12:00.147639 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.147640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.147649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.147705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.148216 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.148876 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.149780 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.150065 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.150136 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.152481 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.152483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.152483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.152490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.154106 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.154490 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.154492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.154492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.154553 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.154749 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.154750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.154751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.154878 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.155051 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.155082 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.155140 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:12:00.156118 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: Bootstrap starting.
I20260924 00:12:00.156958 13897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:00.157255 13897 log.cc:824] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:00.157965 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: No bootstrap required, opened a new log
I20260924 00:12:00.159415 13897 raft_consensus.cc:348] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:12:00.159561 13897 raft_consensus.cc:374] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:00.159621 13897 raft_consensus.cc:729] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62f084bc564928b0895f5aa242a8e6, State: Initialized, Role: FOLLOWER
I20260924 00:12:00.159760 13897 consensus_queue.cc:260] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.160032 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.160087 13897 sys_catalog.cc:564] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:00.160110 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:12:00.162568 13914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:12:00.162622 13914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:12:00.163082 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.155845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.0.98.61:41163)
0924 00:12:00.155888 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.155890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.155900 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:12:00.155925 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.155926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.155935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.155990 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.156425 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.156885 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.157622 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.157817 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.157886 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.160080 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.160081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.160082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.160088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.161535 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.161931 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.161932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.161933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.161998 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.162248 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.162249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.162249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.162379 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.162541 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.162913 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.162983 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:12:00.164435 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:12:00.162791 13829 master_runner.cc:186] Error getting master registration for 127.0.98.62:42461: OK, instance_id { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" instance_seqno: 1790208719781520 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:12:00.165413 13829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:12:00.174571 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.166236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.0.98.60:34457)
0924 00:12:00.166284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.166286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.166354 (+ 68us) server_negotiation.cc:407] Connection header received
0924 00:12:00.166453 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.166454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.166464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.166493 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.167309 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.168048 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169159 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.169445 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169486 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.171295 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.171296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.171297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.171303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.173111 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.173582 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.173584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.173584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.173638 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.173865 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.173866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.173866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.174054 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.174142 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.174368 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.174453 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:12:00.174599 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.165922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.98.61:41163)
0924 00:12:00.165957 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.165959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.166196 (+ 237us) server_negotiation.cc:407] Connection header received
0924 00:12:00.166444 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.166445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.166464 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.166494 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.167354 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.168090 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169190 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.169491 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169534 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.171339 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.171341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.171341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.171349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.173211 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.173670 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.173671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.173671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.173731 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.173964 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.173965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.173965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.174120 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.174215 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.174419 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.174511 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:12:00.175343 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.165934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.0.98.62:42461)
0924 00:12:00.165974 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.165976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.166284 (+ 308us) server_negotiation.cc:407] Connection header received
0924 00:12:00.166380 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.166381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.166393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.166442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.167305 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.168048 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169159 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.169449 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.169486 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.171427 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.171428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.171428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.171436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.173936 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.174320 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.174322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.174322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.174378 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.174571 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.174571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.174572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.174699 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.174768 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.175128 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.175202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:12:00.179893 13822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 3
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:12:00.187016 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.180165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43908)
0924 00:12:00.180221 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.180234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.180285 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.180385 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.180386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.180559 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.180601 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.180603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.181582 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.181583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.182151 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.182153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.182599 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.182599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.182635 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.183052 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.183057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.184390 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.186124 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.186124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.186125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.186289 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.186483 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.186483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.186483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.186531 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.186772 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.186772 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.186808 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.186856 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.186948 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:12:00.187094 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.180239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.98.60:34457)
0924 00:12:00.180286 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.180288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.180297 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:00.180328 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.180328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.180336 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.180380 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.180754 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.181492 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.182222 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.182531 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.182603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.184477 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.184478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.184479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.184486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.186041 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.186357 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.186358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.186358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.186418 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.186581 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.186581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.186582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.186708 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.186766 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.186925 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.187002 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:12:00.188753 13822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: Bootstrap starting.
I20260924 00:12:00.189435 13822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:00.189740 13822 log.cc:824] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:00.190656 13822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: No bootstrap required, opened a new log
I20260924 00:12:00.192258 13822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.192377 13822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:00.192401 13822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bead804e7b94fc9b68f7eb34b678871, State: Initialized, Role: FOLLOWER
I20260924 00:12:00.192579 13822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.192847 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.192911 13822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:00.192934 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:12:00.193200 13775 tablet.cc:2369] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:12:00.195345 13935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:12:00.195405 13935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:12:00.195102 13754 master_runner.cc:186] Error getting master registration for 127.0.98.62:42461: OK, instance_id { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" instance_seqno: 1790208719781520 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:12:00.197424 13754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:12:00.224344 13747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } attempt: 3
I20260924 00:12:00.227051 13924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:00.227180 13924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.227507 13924 leader_election.cc:288] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50cc40ece84049619100c32fe22e40ca (127.0.98.62:42461), de62f084bc564928b0895f5aa242a8e6 (127.0.98.60:34457)
I20260924 00:12:00.227852 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bead804e7b94fc9b68f7eb34b678871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62f084bc564928b0895f5aa242a8e6" is_pre_election: true
I20260924 00:12:00.228012 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bead804e7b94fc9b68f7eb34b678871 in term 0.
I20260924 00:12:00.228236 13773 leader_election.cc:302] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bead804e7b94fc9b68f7eb34b678871, de62f084bc564928b0895f5aa242a8e6; no voters:
I20260924 00:12:00.228377 13924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:12:00.228417 13924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:12:00.228438 13924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:00.229346 13924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.229491 13924 leader_election.cc:288] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [CANDIDATE]: Term 1 election: Requested vote from peers 50cc40ece84049619100c32fe22e40ca (127.0.98.62:42461), de62f084bc564928b0895f5aa242a8e6 (127.0.98.60:34457)
I20260924 00:12:00.229884 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bead804e7b94fc9b68f7eb34b678871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62f084bc564928b0895f5aa242a8e6"
I20260924 00:12:00.229979 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:12:00 dist-test-slave-3q8k krb5kdc[13666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:12:00.230850 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bead804e7b94fc9b68f7eb34b678871 in term 1.
I20260924 00:12:00.231007 13773 leader_election.cc:302] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bead804e7b94fc9b68f7eb34b678871, de62f084bc564928b0895f5aa242a8e6; no voters:
I20260924 00:12:00.231107 13924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:00.231302 13924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [term 1 LEADER]: Becoming Leader. State: Replica: 5bead804e7b94fc9b68f7eb34b678871, State: Running, Role: LEADER
I20260924 00:12:00.231401 13924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.231889 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bead804e7b94fc9b68f7eb34b678871. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.231966 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:00.232178 13939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:00.232373 13939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:00.234297 13871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [term 1 FOLLOWER]: Refusing update from remote peer 5bead804e7b94fc9b68f7eb34b678871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:00.234581 13924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:00.235762 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.224657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34457 (local address 127.0.0.1:43920)
0924 00:12:00.224709 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:00.224721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:00.224759 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:00.225036 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:00.225037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:00.225235 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:00.225296 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.225298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.226234 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.226236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.227060 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:00.227062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.228413 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.228414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:00.228481 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.229017 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:00.229034 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:00.230575 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:00.233111 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.233112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.233113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.233354 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.233648 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:00.233649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:00.233649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:00.233731 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:00.235393 (+ 1662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:00.235394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:00.235476 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 00:12:00.235550 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 00:12:00.235661 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:12:00.235764 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bead804e7b94fc9b68f7eb34b678871. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.235842 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:12:00.236449 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:00.224837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.98.60:34457)
0924 00:12:00.224897 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:00.224899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:00.224908 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:00.224935 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:00.224936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:00.224948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:00.225018 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:00.225377 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.226144 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.227157 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:00.228296 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:00.228681 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:00.230667 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:00.230668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:00.230669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:00.230677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:00.232888 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.233473 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.233475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.233475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.233553 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:00.233819 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:00.233820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:00.233821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:00.233976 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:00.234042 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:00.236282 (+ 2240us) server_negotiation.cc:299] Negotiation successful
0924 00:12:00.236366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:12:00.236678 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.236765 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:00.236960 13940 mvcc.cc:205] Tried to move back new op lower bound from 7332694918077173760 to 7332694918068740096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694918077173760}]
I20260924 00:12:00.237672 13939 catalog_manager.cc:1347] Generated new cluster ID: a7d4a9d3a29b4e88badbc3c053b5a164
I20260924 00:12:00.237709 13939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:12:00.237989 13747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: Bootstrap starting.
I20260924 00:12:00.238709 13747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:00.238981 13747 log.cc:824] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:00.240578 13747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: No bootstrap required, opened a new log
I20260924 00:12:00.242554 13747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.242682 13747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:00.242717 13747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cc40ece84049619100c32fe22e40ca, State: Initialized, Role: FOLLOWER
I20260924 00:12:00.242830 13747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } }
I20260924 00:12:00.243116 13747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:00.243259 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bead804e7b94fc9b68f7eb34b678871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50cc40ece84049619100c32fe22e40ca" is_pre_election: true
I20260924 00:12:00.243263 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.243347 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:12:00.243366 13721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bead804e7b94fc9b68f7eb34b678871 in term 0.
I20260924 00:12:00.243521 13719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:00.244364 13719 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 1 FOLLOWER]: Refusing update from remote peer 5bead804e7b94fc9b68f7eb34b678871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:00.244513 13720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bead804e7b94fc9b68f7eb34b678871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50cc40ece84049619100c32fe22e40ca"
I20260924 00:12:00.245200 13720 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bead804e7b94fc9b68f7eb34b678871 in term 1.
I20260924 00:12:00.246316 13924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:00.249516 13941 mvcc.cc:205] Tried to move back new op lower bound from 7332694918077173760 to 7332694918068740096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694918077173760 or (T in {7332694918077173760})}]
I20260924 00:12:00.249818 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.250000 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:12:00.250095 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bead804e7b94fc9b68f7eb34b678871. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.250202 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:12:00.250276 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.250373 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:12:00.250689 13939 catalog_manager.cc:1370] Generated new certificate authority record
W20260924 00:12:00.251322 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 00:12:00.251425 13939 catalog_manager.cc:1504] Loading token signing keys...
W20260924 00:12:00.251474 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:12:00.252182 13948 mvcc.cc:205] Tried to move back new op lower bound from 7332694918077173760 to 7332694918068740096. Current Snapshot: MvccSnapshot[applied={T|T < 7332694918077173760}]
I20260924 00:12:00.252862 13924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bead804e7b94fc9b68f7eb34b678871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cc40ece84049619100c32fe22e40ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42461 } } peers { permanent_uuid: "5bead804e7b94fc9b68f7eb34b678871" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41163 } } peers { permanent_uuid: "de62f084bc564928b0895f5aa242a8e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34457 } } }
I20260924 00:12:00.252946 13924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:00.258423 13939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bead804e7b94fc9b68f7eb34b678871: Generated new TSK 0
I20260924 00:12:00.258602 13939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:12:01.163053 13914 catalog_manager.cc:1259] Loaded cluster ID: a7d4a9d3a29b4e88badbc3c053b5a164
I20260924 00:12:01.163136 13914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: loading cluster ID for follower catalog manager: success
I20260924 00:12:01.164029 13914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: acquiring CA information for follower catalog manager: success
I20260924 00:12:01.164340 13914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de62f084bc564928b0895f5aa242a8e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:12:01.252107 13961 catalog_manager.cc:1259] Loaded cluster ID: a7d4a9d3a29b4e88badbc3c053b5a164
I20260924 00:12:01.252187 13961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: loading cluster ID for follower catalog manager: success
I20260924 00:12:01.253036 13961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: acquiring CA information for follower catalog manager: success
I20260924 00:12:01.253340 13961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50cc40ece84049619100c32fe22e40ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:12:01.323992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13679
I20260924 00:12:01.329308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13754
I20260924 00:12:01.334367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13829
2026-09-24T00:12:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2833 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:12:01 dist-test-slave-3q8k krb5kdc[13974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:12:01 dist-test-slave-3q8k krb5kdc[13974](info): set up 2 sockets
Sep 24 00:12:01 dist-test-slave-3q8k krb5kdc[13974](info): commencing operation
krb5kdc: starting...
W20260924 00:12:02.371590 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:12:02.490209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:02.569806 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:02.570024 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:02.570045 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:02.571170 13987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:02.571236 13987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:02.571252 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:02.571265 13987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:02.571277 13987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:02.572664 13987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:02.572860 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:02.573040 13987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:02.577739 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:02.577729 13995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:02.577729 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:02.577962 13987 server_base.cc:1034] running on GCE node
I20260924 00:12:02.578364 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:02.578655 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:02.579816 13987 hybrid_clock.cc:648] HybridClock initialized: now 1790208722579811 us; error 39 us; skew 500 ppm
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:02.582230 13987 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:02.582692 13987 webserver.cc:460] Webserver started at http://127.0.98.62:42269/ using document root <none> and password file <none>
I20260924 00:12:02.582912 13987 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:02.582976 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:02.583122 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:02.584204 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "8d3806f25c2140a48c59bcf0fcbd48d1"
format_stamp: "Formatted at 2026-09-24 00:12:02 on dist-test-slave-3q8k"
server_key: "6934439e0123db798806ca7584ed9074"
server_key_iv: "39dc57e27d3bb6c37a1e796773830495"
server_key_version: "encryptionkey@0"
I20260924 00:12:02.584601 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d3806f25c2140a48c59bcf0fcbd48d1"
format_stamp: "Formatted at 2026-09-24 00:12:02 on dist-test-slave-3q8k"
server_key: "6934439e0123db798806ca7584ed9074"
server_key_iv: "39dc57e27d3bb6c37a1e796773830495"
server_key_version: "encryptionkey@0"
I20260924 00:12:02.585881 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:12:02.586562 14002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:02.586725 13987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:02.586819 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8d3806f25c2140a48c59bcf0fcbd48d1"
format_stamp: "Formatted at 2026-09-24 00:12:02 on dist-test-slave-3q8k"
server_key: "6934439e0123db798806ca7584ed9074"
server_key_iv: "39dc57e27d3bb6c37a1e796773830495"
server_key_version: "encryptionkey@0"
I20260924 00:12:02.586899 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:02.602775 13987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:02.604775 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:02.604920 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:02.608872 13987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40689
I20260924 00:12:02.608922 14054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40689 every 8 connection(s)
I20260924 00:12:02.609280 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:02.610142 14055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:02.612547 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: Bootstrap starting.
I20260924 00:12:02.613140 14055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:02.613437 14055 log.cc:824] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:02.614198 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: No bootstrap required, opened a new log
I20260924 00:12:02.615041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 13987
I20260924 00:12:02.615130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:12:02.615325 392 external_mini_cluster.cc:1428] Setting key 431e69b42b09f153a22ce05faec7ba5e
I20260924 00:12:02.616288 14055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } }
I20260924 00:12:02.616462 14055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:02.616567 14055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3806f25c2140a48c59bcf0fcbd48d1, State: Initialized, Role: FOLLOWER
I20260924 00:12:02.616724 14055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } }
I20260924 00:12:02.616811 14055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:02.616848 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:02.616897 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:02.617974 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } }
I20260924 00:12:02.618080 14055 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3806f25c2140a48c59bcf0fcbd48d1; no voters:
I20260924 00:12:02.618225 14055 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:12:02.618278 14060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:02.618443 14060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3806f25c2140a48c59bcf0fcbd48d1, State: Running, Role: LEADER
I20260924 00:12:02.618450 14055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:02.619200 14060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } }
I20260924 00:12:02.619889 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } } }
I20260924 00:12:02.620018 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:02.619922 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3806f25c2140a48c59bcf0fcbd48d1. Latest consensus state: current_term: 1 leader_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3806f25c2140a48c59bcf0fcbd48d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40689 } } }
I20260924 00:12:02.620177 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1 [sys.catalog]: This master's current role is: LEADER
W20260924 00:12:02.620460 14074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:12:02.620564 14074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:12:02.620666 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:02.620877 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:02.622735 14075 catalog_manager.cc:1347] Generated new cluster ID: 3c2cdbd3cad9420185f890ca3cb29022
I20260924 00:12:02.622777 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:02.627748 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:02.615836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.0.98.62:40689)
0924 00:12:02.616039 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:02.616051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:02.616091 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:12:02.616245 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:02.616253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:02.616310 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:02.616452 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:02.617253 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:02.618659 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:02.619878 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:02.620455 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:02.620600 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:02.623459 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:02.623463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:02.623463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:02.623475 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:02.625467 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:02.625879 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:02.625883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:02.625883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:02.626028 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:02.626235 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:02.626236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:02.626236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:02.626546 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:02.626698 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:02.627286 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 00:12:02.627475 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260924 00:12:02.628774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:12:02.638708 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:12:02.639488 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:12:02.652171 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3806f25c2140a48c59bcf0fcbd48d1: Generated new TSK 0
I20260924 00:12:02.652419 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:12:02.682866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 13987
2026-09-24T00:12:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[14082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[14082](info): set up 2 sockets
Sep 24 00:12:02 dist-test-slave-3q8k krb5kdc[14082](info): commencing operation
krb5kdc: starting...
W20260924 00:12:03.751235 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:12:03 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:12:03.870965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:03.951452 14095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:03.951694 14095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:03.951730 14095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:03.952855 14095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:03.952921 14095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:03.952944 14095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:03.952966 14095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:03.952993 14095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:03.954412 14095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:03.954672 14095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:03.954895 14095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:03.959589 14103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:03.959745 14095 server_base.cc:1034] running on GCE node
W20260924 00:12:03.959618 14100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:03.959580 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:03.960067 14095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:03.960351 14095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:03.961508 14095 hybrid_clock.cc:648] HybridClock initialized: now 1790208723961499 us; error 39 us; skew 500 ppm
Sep 24 00:12:03 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:03.963989 14095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:03.964421 14095 webserver.cc:460] Webserver started at http://127.0.98.62:39811/ using document root <none> and password file <none>
I20260924 00:12:03.964640 14095 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:03.964705 14095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:03.964848 14095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:03.965916 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82"
format_stamp: "Formatted at 2026-09-24 00:12:03 on dist-test-slave-3q8k"
server_key: "0bb04e6c42076c5cd16f2755f803a939"
server_key_iv: "2decabbd816f6657f10bb495c1fe1e75"
server_key_version: "encryptionkey@0"
I20260924 00:12:03.966287 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82"
format_stamp: "Formatted at 2026-09-24 00:12:03 on dist-test-slave-3q8k"
server_key: "0bb04e6c42076c5cd16f2755f803a939"
server_key_iv: "2decabbd816f6657f10bb495c1fe1e75"
server_key_version: "encryptionkey@0"
I20260924 00:12:03.967658 14095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:12:03.968428 14110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:03.968588 14095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:03.968685 14095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82"
format_stamp: "Formatted at 2026-09-24 00:12:03 on dist-test-slave-3q8k"
server_key: "0bb04e6c42076c5cd16f2755f803a939"
server_key_iv: "2decabbd816f6657f10bb495c1fe1e75"
server_key_version: "encryptionkey@0"
I20260924 00:12:03.968761 14095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:03.986653 14095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:03.988741 14095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:03.988912 14095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:03.992810 14095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38687
I20260924 00:12:03.992846 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38687 every 8 connection(s)
I20260924 00:12:03.993232 14095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:03.994166 14163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:03.995640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14095
I20260924 00:12:03.995764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:12:03.995942 392 external_mini_cluster.cc:1428] Setting key 219a6446682d4676fb450d7fd2298313
I20260924 00:12:03.996727 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap starting.
I20260924 00:12:03.997289 14163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:03.997586 14163 log.cc:824] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:03.998267 14163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: No bootstrap required, opened a new log
I20260924 00:12:04.000022 14163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.000175 14163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:04.000221 14163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.000416 14163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.000511 14163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:04.000548 14163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:04.000607 14163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:04.001317 14163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.001428 14163 leader_election.cc:302] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e346a7f6e5a4d73b5dc09a1f19ade82; no voters:
I20260924 00:12:04.001608 14163 leader_election.cc:288] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:12:04.001648 14168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:04.001798 14168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 LEADER]: Becoming Leader. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Running, Role: LEADER
I20260924 00:12:04.001886 14163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:04.001921 14168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.002305 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e346a7f6e5a4d73b5dc09a1f19ade82. Latest consensus state: current_term: 1 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 00:12:04.002532 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:04.002445 14170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
Sep 24 00:12:03 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:04.002902 14170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:04.003031 14177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:04.003573 14177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:04.005311 14177 catalog_manager.cc:1347] Generated new cluster ID: 3c94c7ef90dc47a58cfbd4fc87bedac3
I20260924 00:12:04.005366 14177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:12:04.007073 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:03.996479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.0.98.62:38687)
0924 00:12:03.996663 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:03.996672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:03.996697 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:12:03.997180 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:03.997188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:03.997236 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:03.997386 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:03.998112 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:03.999034 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:03.999852 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.000289 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.000397 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.003611 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.003616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.003616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.003629 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.005628 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.005974 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.005977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.005978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.006065 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.006257 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.006258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.006258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.006489 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.006621 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.006738 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.006885 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 00:12:04.052273 14177 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:12:04.052834 14177 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:12:04.054880 14177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Generated new TSK 0
I20260924 00:12:04.055063 14177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:12:04.088240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:04.183009 14189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.183255 14189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.183282 14189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.184393 14189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.184432 14189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.184460 14189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:12:04.185861 14189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.186041 14189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.186223 14189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.190939 14195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.191004 14194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.190944 14197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.191357 14189 server_base.cc:1034] running on GCE node
I20260924 00:12:04.191622 14189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.191865 14189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.193025 14189 hybrid_clock.cc:648] HybridClock initialized: now 1790208724193010 us; error 40 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.195547 14189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:12:04.196035 14189 webserver.cc:460] Webserver started at http://127.0.98.1:33289/ using document root <none> and password file <none>
I20260924 00:12:04.196262 14189 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.196336 14189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.196478 14189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:04.197532 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "c48da81c67ab4522823cb79da7239160"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "4b2e36e3cf2569355dbfdb4d93d20a7b"
server_key_iv: "937466148820d4a1ccdff6421e6468f1"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.197902 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c48da81c67ab4522823cb79da7239160"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "4b2e36e3cf2569355dbfdb4d93d20a7b"
server_key_iv: "937466148820d4a1ccdff6421e6468f1"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.199257 14189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:12:04.200027 14204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.200193 14189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:04.200289 14189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "c48da81c67ab4522823cb79da7239160"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "4b2e36e3cf2569355dbfdb4d93d20a7b"
server_key_iv: "937466148820d4a1ccdff6421e6468f1"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.200394 14189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:04.212561 14189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:04.214740 14189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:04.214905 14189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:04.215139 14189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:04.215490 14189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:04.215544 14189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.215587 14189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:04.215611 14189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.221385 14189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42003
I20260924 00:12:04.221423 14317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42003 every 8 connection(s)
I20260924 00:12:04.221767 14189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:12:04.223634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14189
I20260924 00:12:04.223754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:12:04.223994 392 external_mini_cluster.cc:1428] Setting key 61041cc9e50f431f7795f167b9f82051
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:12:04.231606 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.222509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36193 (local address 127.0.98.62:38687)
0924 00:12:04.222567 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.222570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.222763 (+ 193us) server_negotiation.cc:407] Connection header received
0924 00:12:04.223319 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.223321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.223333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.223395 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.224597 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.225104 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.226386 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.226597 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.226651 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.228792 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.228794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.228794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.228803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.230162 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.230586 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.230587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.230588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.230653 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.230905 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.230906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.230906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.231027 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.231101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.231419 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.231507 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:12:04.231835 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.222491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:36193)
0924 00:12:04.222650 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.222684 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.223184 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.223481 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.223484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.224243 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:04.224519 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.224525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.225211 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.225212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.226281 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.226284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.226680 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.226681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.226824 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.227357 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:04.227372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:04.228698 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:04.230261 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.230268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.230270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.230497 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.230732 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.230732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.230733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.230841 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.231092 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:04.231098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:04.231284 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.231400 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.231569 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260924 00:12:04.232317 14318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
I20260924 00:12:04.232399 14318 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:04.232611 14318 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:04.233068 14127 ts_manager.cc:194] Registered new tserver with Master: c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003)
I20260924 00:12:04.233729 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:12:04.250200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:04.330428 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.330655 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.330693 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.331880 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.331939 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.331985 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:12:04.333432 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.333703 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.333925 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.338963 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.338954 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.338997 14323 server_base.cc:1034] running on GCE node
W20260924 00:12:04.338965 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.339356 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.339564 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.340699 14323 hybrid_clock.cc:648] HybridClock initialized: now 1790208724340687 us; error 28 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.343146 14323 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:12:04.343616 14323 webserver.cc:460] Webserver started at http://127.0.98.2:39483/ using document root <none> and password file <none>
I20260924 00:12:04.343847 14323 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.343914 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.344055 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:04.345101 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "739f976789ae4192b64960f1d247b7f1"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "ece49261838b64dc40329e0bc9e8978b"
server_key_iv: "c4484501bdf11ce8f3400e13e4f15cf3"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.345448 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "739f976789ae4192b64960f1d247b7f1"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "ece49261838b64dc40329e0bc9e8978b"
server_key_iv: "c4484501bdf11ce8f3400e13e4f15cf3"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.346725 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:12:04.347460 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.347621 14323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:04.347719 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "739f976789ae4192b64960f1d247b7f1"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "ece49261838b64dc40329e0bc9e8978b"
server_key_iv: "c4484501bdf11ce8f3400e13e4f15cf3"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.347788 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:04.363143 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:04.365494 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:04.365660 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:04.365878 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:04.366175 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:04.366225 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.366264 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:04.366291 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.372185 14323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43385
I20260924 00:12:04.372226 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43385 every 8 connection(s)
I20260924 00:12:04.372598 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:12:04.374696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14323
I20260924 00:12:04.374809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:12:04.375010 392 external_mini_cluster.cc:1428] Setting key c6ceb84ba9a14ef66a18b421e3c2bda1
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:04.383447 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.373266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60139 (local address 127.0.98.62:38687)
0924 00:12:04.373312 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.373314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.373558 (+ 244us) server_negotiation.cc:407] Connection header received
0924 00:12:04.374038 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.374040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.374050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.374101 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.375348 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.376063 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.377503 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.377715 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.377801 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.380273 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.380275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.380275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.380285 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.382053 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.382442 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.382443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.382444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.382508 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.382743 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.382744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.382744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.382887 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.382958 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.383241 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.383327 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:12:04.383562 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.373258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:60139)
0924 00:12:04.373406 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.373442 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.373942 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.374156 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.374159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.374890 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:04.375205 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.375211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.376156 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.376157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.377412 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.377415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.377811 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.377812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.377929 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.378489 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:04.378499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:04.380174 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:04.382134 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.382140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.382142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.382367 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.382587 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.382587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.382588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.382679 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.382960 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:04.382963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:04.383085 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.383163 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.383365 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 00:12:04.384020 14452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:12:04.384133 14452 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:04.384311 14452 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:04.384672 14127 ts_manager.cc:194] Registered new tserver with Master: 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
I20260924 00:12:04.385011 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:12:04.402133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:04.483457 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.483668 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.483691 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.484906 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.484953 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.484982 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:12:04.486515 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.486740 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.486964 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.491981 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.492100 14457 server_base.cc:1034] running on GCE node
W20260924 00:12:04.492007 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.492007 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.492480 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.492730 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.493885 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790208724493916 us; error 68 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.496377 14457 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:12:04.496816 14457 webserver.cc:460] Webserver started at http://127.0.98.3:39791/ using document root <none> and password file <none>
I20260924 00:12:04.497052 14457 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.497116 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.497259 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:04.498293 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e49aa0f487147a89d477d8dfcec2e44"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "f346b8900aa4695a334c79fe822f3df4"
server_key_iv: "5b8c51fcca2474decee2c9ad8905245c"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.498656 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e49aa0f487147a89d477d8dfcec2e44"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "f346b8900aa4695a334c79fe822f3df4"
server_key_iv: "5b8c51fcca2474decee2c9ad8905245c"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.500054 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:12:04.500766 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.500926 14457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:04.501019 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "3e49aa0f487147a89d477d8dfcec2e44"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "f346b8900aa4695a334c79fe822f3df4"
server_key_iv: "5b8c51fcca2474decee2c9ad8905245c"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.501094 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:04.512233 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:04.514525 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:04.514638 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:04.514868 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:04.515195 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:04.515280 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.515331 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:04.515358 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.521288 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44261
I20260924 00:12:04.521370 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44261 every 8 connection(s)
I20260924 00:12:04.521688 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:12:04.526952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14457
I20260924 00:12:04.527060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:12:04.527283 392 external_mini_cluster.cc:1428] Setting key d96c92ba208e4370196653d4a80517de
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:04.530956 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.522421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40193 (local address 127.0.98.62:38687)
0924 00:12:04.522483 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.522485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.522703 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:12:04.523186 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.523187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.523196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.523301 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.524222 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.524715 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.525686 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.525866 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.525918 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.528209 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.528210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.528211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.528219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.529595 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.530016 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.530017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.530018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.530074 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.530315 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.530316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.530316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.530437 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.530510 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.530780 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.530858 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:12:04.531167 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.522413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:40193)
0924 00:12:04.522576 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.522619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.523104 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.523372 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.523376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.523940 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:04.524144 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.524149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.524796 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.524797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.525612 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.525614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.525937 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.525938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.526043 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.526501 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:04.526509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:04.528111 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:04.529696 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.529704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.529707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.529927 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.530136 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.530137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.530138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.530248 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.530501 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:04.530514 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:04.530651 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.530760 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.530924 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":106,"threads_started":1}
I20260924 00:12:04.531661 14586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
I20260924 00:12:04.531773 14586 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:04.531955 14586 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:04.532313 14127 ts_manager.cc:194] Registered new tserver with Master: 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261)
I20260924 00:12:04.532696 14127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40193
I20260924 00:12:04.538894 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:12:04.545912 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.540242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.0.98.62:38687)
0924 00:12:04.540308 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.540310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.540397 (+ 87us) server_negotiation.cc:407] Connection header received
0924 00:12:04.540441 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.540442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.540451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.540499 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.541049 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.541521 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.542211 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.542387 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.542435 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.543617 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.543618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.543619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.543625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.544960 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.545257 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.545258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.545258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.545306 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.545446 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.545446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.545446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.545583 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.545657 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.545739 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.545800 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:12:04.547390 14127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:12:04.549368 14127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:12:04.555771 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:43498)
0924 00:12:04.552015 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.552026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.552089 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.552359 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.552360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.552362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:04.552415 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.552417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553304 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.553305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.554545 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.554547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555458 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.555458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.555521 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.555532 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.555612 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.555663 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":38,"threads_started":1}
I20260924 00:12:04.555869 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.98.3:44261)
0924 00:12:04.552024 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.552030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.552043 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:12:04.552083 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.552086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.552102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.552204 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:04.552375 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553524 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.554759 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555510 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.555588 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.555646 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.555737 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.555776 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260924 00:12:04.555862 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44261 (local address 127.0.0.1:40388)
0924 00:12:04.551606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.551618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.551676 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.552210 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.552213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.552215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:04.552287 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.552289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553623 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.553624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.554676 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.554678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555593 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.555594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.555652 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.555655 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.555721 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.555758 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260924 00:12:04.556052 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43385 (local address 127.0.0.1:56706)
0924 00:12:04.551566 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.551611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.551677 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.552523 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.552524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.552526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:04.552595 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.552596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553839 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.553840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.554813 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.554816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555715 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.555716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.555778 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.555782 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.555815 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.555853 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 00:12:04.556159 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.1:42003)
0924 00:12:04.551975 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.551979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.552097 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:12:04.552169 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.552172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.552188 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.552265 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:04.552490 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553204 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.554634 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555301 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.555367 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.555429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.555969 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.556014 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260924 00:12:04.556474 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.551827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.98.2:43385)
0924 00:12:04.552347 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.552351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.552360 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:04.552397 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.552399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.552416 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.552671 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:04.552739 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.553742 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.554902 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.555614 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.556172 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.556292 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.556317 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.556362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":87,"threads_started":1}
I20260924 00:12:04.556895 14252 tablet_service.cc:1467] Processing CreateTablet for tablet 1709392914be47a69794a1ca5c410725 (DEFAULT_TABLE table=test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:04.556907 14520 tablet_service.cc:1467] Processing CreateTablet for tablet 1709392914be47a69794a1ca5c410725 (DEFAULT_TABLE table=test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:04.557248 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1709392914be47a69794a1ca5c410725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:04.557252 14520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1709392914be47a69794a1ca5c410725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:04.557267 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 1709392914be47a69794a1ca5c410725 (DEFAULT_TABLE table=test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:04.557559 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1709392914be47a69794a1ca5c410725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:04.559930 14600 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Bootstrap starting.
I20260924 00:12:04.560509 14601 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Bootstrap starting.
I20260924 00:12:04.560710 14600 tablet_bootstrap.cc:654] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:04.560827 14602 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Bootstrap starting.
I20260924 00:12:04.561205 14601 tablet_bootstrap.cc:654] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:04.561277 14600 log.cc:824] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:04.561384 14602 tablet_bootstrap.cc:654] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:04.561760 14601 log.cc:824] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:04.561975 14602 log.cc:824] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:04.562134 14600 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: No bootstrap required, opened a new log
I20260924 00:12:04.562222 14600 ts_tablet_manager.cc:1397] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:04.562526 14601 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: No bootstrap required, opened a new log
I20260924 00:12:04.562592 14601 ts_tablet_manager.cc:1397] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:04.562649 14602 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: No bootstrap required, opened a new log
I20260924 00:12:04.562713 14602 ts_tablet_manager.cc:1397] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:12:04.563974 14600 raft_consensus.cc:348] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.564133 14600 raft_consensus.cc:374] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:04.564177 14600 raft_consensus.cc:729] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48da81c67ab4522823cb79da7239160, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.564311 14600 consensus_queue.cc:260] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.564574 14600 ts_tablet_manager.cc:1428] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:04.564553 14601 raft_consensus.cc:348] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.564570 14602 raft_consensus.cc:348] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.564715 14318 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
I20260924 00:12:04.564730 14602 raft_consensus.cc:374] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:04.564730 14601 raft_consensus.cc:374] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:04.564765 14601 raft_consensus.cc:729] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e49aa0f487147a89d477d8dfcec2e44, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.564765 14602 raft_consensus.cc:729] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739f976789ae4192b64960f1d247b7f1, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.564872 14602 consensus_queue.cc:260] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.564872 14601 consensus_queue.cc:260] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.565124 14602 ts_tablet_manager.cc:1428] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:12:04.565243 14452 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
I20260924 00:12:04.565474 14586 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
I20260924 00:12:04.565702 14601 ts_tablet_manager.cc:1428] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:12:04.573554 14606 raft_consensus.cc:482] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:04.573678 14606 raft_consensus.cc:504] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.574040 14606 leader_election.cc:288] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385), 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261)
I20260924 00:12:04.576802 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.574290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35829 (local address 127.0.98.2:43385)
0924 00:12:04.574326 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.574328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.574335 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:12:04.574359 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.574360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.574372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.574437 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:04.574604 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.575120 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576090 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.576462 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576516 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.576530 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.576683 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.576730 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:12:04.576802 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.574172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43385 (local address 127.0.98.1:35829)
0924 00:12:04.574220 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.574232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.574277 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.574441 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.574442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.574443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:04.574536 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.574538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.575195 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.575196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.576005 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.576008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.576538 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.576617 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.576620 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.576668 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.576699 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":21}
I20260924 00:12:04.576856 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.574238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51941 (local address 127.0.98.3:44261)
0924 00:12:04.574282 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.574284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.574474 (+ 190us) server_negotiation.cc:407] Connection header received
0924 00:12:04.574511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.574512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.574524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.574582 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:04.574719 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.575169 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576130 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.576482 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576571 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.576580 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.576739 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.576776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:12:04.576959 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.574174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44261 (local address 127.0.98.1:51941)
0924 00:12:04.574402 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.574413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.574468 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.574590 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.574591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.574592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:04.574661 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.574663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.575263 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.575263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.576019 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.576028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.576583 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.576584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.576664 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.576668 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.576726 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.576760 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":57,"thread_start_us":101,"threads_started":1}
I20260924 00:12:04.577293 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1709392914be47a69794a1ca5c410725" candidate_uuid: "c48da81c67ab4522823cb79da7239160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e49aa0f487147a89d477d8dfcec2e44" is_pre_election: true
I20260924 00:12:04.577404 14540 raft_consensus.cc:2454] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48da81c67ab4522823cb79da7239160 in term 0.
I20260924 00:12:04.577484 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1709392914be47a69794a1ca5c410725" candidate_uuid: "c48da81c67ab4522823cb79da7239160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f976789ae4192b64960f1d247b7f1" is_pre_election: true
I20260924 00:12:04.577584 14206 leader_election.cc:302] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e49aa0f487147a89d477d8dfcec2e44, c48da81c67ab4522823cb79da7239160; no voters:
I20260924 00:12:04.577618 14406 raft_consensus.cc:2454] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48da81c67ab4522823cb79da7239160 in term 0.
I20260924 00:12:04.577692 14606 raft_consensus.cc:2781] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:12:04.577754 14606 raft_consensus.cc:482] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:12:04.577787 14606 raft_consensus.cc:3037] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:04.578466 14606 raft_consensus.cc:504] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.578617 14606 leader_election.cc:288] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 1 election: Requested vote from peers 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385), 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261)
I20260924 00:12:04.578761 14406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1709392914be47a69794a1ca5c410725" candidate_uuid: "c48da81c67ab4522823cb79da7239160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f976789ae4192b64960f1d247b7f1"
I20260924 00:12:04.578794 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1709392914be47a69794a1ca5c410725" candidate_uuid: "c48da81c67ab4522823cb79da7239160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e49aa0f487147a89d477d8dfcec2e44"
I20260924 00:12:04.578841 14406 raft_consensus.cc:3037] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:04.578878 14540 raft_consensus.cc:3037] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:04.579771 14540 raft_consensus.cc:2454] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48da81c67ab4522823cb79da7239160 in term 1.
I20260924 00:12:04.579771 14406 raft_consensus.cc:2454] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48da81c67ab4522823cb79da7239160 in term 1.
I20260924 00:12:04.579941 14206 leader_election.cc:302] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e49aa0f487147a89d477d8dfcec2e44, c48da81c67ab4522823cb79da7239160; no voters:
I20260924 00:12:04.580035 14606 raft_consensus.cc:2781] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:04.580192 14606 raft_consensus.cc:686] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 LEADER]: Becoming Leader. State: Replica: c48da81c67ab4522823cb79da7239160, State: Running, Role: LEADER
I20260924 00:12:04.580291 14606 consensus_queue.cc:237] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.581060 14127 catalog_manager.cc:5495] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 reported cstate change: term changed from 0 to 1, leader changed from <none> to c48da81c67ab4522823cb79da7239160 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c48da81c67ab4522823cb79da7239160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260924 00:12:04.590075 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.587935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.0.98.1:42003)
0924 00:12:04.587981 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.587983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.588040 (+ 57us) server_negotiation.cc:407] Connection header received
0924 00:12:04.588111 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.588112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.588128 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.588176 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:04.588287 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.588731 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.589418 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.589599 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.589649 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.589761 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:04.589853 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:12:04.589907 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.589937 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.589952 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:12:04.591833 14540 raft_consensus.cc:1261] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Refusing update from remote peer c48da81c67ab4522823cb79da7239160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:04.591897 14406 raft_consensus.cc:1261] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Refusing update from remote peer c48da81c67ab4522823cb79da7239160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:04.592178 14606 consensus_queue.cc:1035] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:04.592329 14610 consensus_queue.cc:1035] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:04.594521 14613 mvcc.cc:205] Tried to move back new op lower bound from 7332694935925366784 to 7332694935881404416. Current Snapshot: MvccSnapshot[applied={T|T < 7332694935925366784}]
I20260924 00:12:04.595124 14615 mvcc.cc:205] Tried to move back new op lower bound from 7332694935925366784 to 7332694935881404416. Current Snapshot: MvccSnapshot[applied={T|T < 7332694935925366784}]
I20260924 00:12:04.596560 14127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:04.596652 14127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:04.596622 14614 mvcc.cc:205] Tried to move back new op lower bound from 7332694935925366784 to 7332694935881404416. Current Snapshot: MvccSnapshot[applied={T|T < 7332694935925366784 or (T in {7332694935925366784})}]
I20260924 00:12:04.597389 14127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Sending DeleteTablet for 3 replicas of tablet 1709392914be47a69794a1ca5c410725
I20260924 00:12:04.597700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14189
I20260924 00:12:04.597698 14520 tablet_service.cc:1514] Processing DeleteTablet for tablet 1709392914be47a69794a1ca5c410725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40388
I20260924 00:12:04.597745 14252 tablet_service.cc:1514] Processing DeleteTablet for tablet 1709392914be47a69794a1ca5c410725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43498
I20260924 00:12:04.598562 14623 tablet_replica.cc:330] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: stopping tablet replica
I20260924 00:12:04.598682 14623 raft_consensus.cc:2229] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:04.598784 14623 raft_consensus.cc:2258] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:04.599162 14623 ts_tablet_manager.cc:1905] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:04.599659 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 1709392914be47a69794a1ca5c410725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56706
I20260924 00:12:04.599871 14624 tablet_replica.cc:330] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: stopping tablet replica
I20260924 00:12:04.600008 14624 raft_consensus.cc:2229] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:04.600158 14624 raft_consensus.cc:2258] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:04.600651 14624 ts_tablet_manager.cc:1905] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:04.601151 14623 ts_tablet_manager.cc:1918] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:04.601230 14623 log.cc:1196] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/1709392914be47a69794a1ca5c410725
I20260924 00:12:04.601575 14623 ts_tablet_manager.cc:1939] T 1709392914be47a69794a1ca5c410725 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting consensus metadata
I20260924 00:12:04.601966 14112 catalog_manager.cc:4841] TS 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): tablet 1709392914be47a69794a1ca5c410725 (table test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]) successfully deleted
I20260924 00:12:04.602499 14624 ts_tablet_manager.cc:1918] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:04.602581 14624 log.cc:1196] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/1709392914be47a69794a1ca5c410725
I20260924 00:12:04.602842 14624 ts_tablet_manager.cc:1939] T 1709392914be47a69794a1ca5c410725 P 739f976789ae4192b64960f1d247b7f1: Deleting consensus metadata
I20260924 00:12:04.603408 14111 catalog_manager.cc:4841] TS 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): tablet 1709392914be47a69794a1ca5c410725 (table test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]) successfully deleted
W20260924 00:12:04.603948 14111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42003 (error 108)
I20260924 00:12:04.604159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14323
I20260924 00:12:04.605068 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.604324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:43522)
0924 00:12:04.604966 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.605006 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605}
W20260924 00:12:04.606554 14111 catalog_manager.cc:4584] TS c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1709392914be47a69794a1ca5c410725: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
I20260924 00:12:04.609760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14457
I20260924 00:12:04.614157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14095
I20260924 00:12:04.619346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--webserver_interface=127.0.98.62
--webserver_port=39811
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:04.700271 14626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.700551 14626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.700587 14626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.701743 14626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:04.701807 14626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.701840 14626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.701870 14626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:04.701893 14626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:04.703370 14626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.703639 14626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.703868 14626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.712538 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.712524 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.712661 14626 server_base.cc:1034] running on GCE node
W20260924 00:12:04.712509 14635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.712972 14626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.713260 14626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.714432 14626 hybrid_clock.cc:648] HybridClock initialized: now 1790208724714417 us; error 33 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.716977 14626 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:04.717391 14626 webserver.cc:460] Webserver started at http://127.0.98.62:39811/ using document root <none> and password file <none>
I20260924 00:12:04.717599 14626 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.717655 14626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.719076 14626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:04.719770 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.719954 14626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.720059 14626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82"
format_stamp: "Formatted at 2026-09-24 00:12:03 on dist-test-slave-3q8k"
server_key: "0bb04e6c42076c5cd16f2755f803a939"
server_key_iv: "2decabbd816f6657f10bb495c1fe1e75"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.720448 14626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:04.734640 14626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:04.737757 14626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:04.737917 14626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:04.742964 14626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38687
I20260924 00:12:04.743024 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38687 every 8 connection(s)
I20260924 00:12:04.743453 14626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:04.743846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14626
I20260924 00:12:04.744306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42003
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=33289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:04.745819 14695 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:12:04.746518 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap starting.
I20260924 00:12:04.754688 14695 log.cc:824] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:04.756592 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:04.756886 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap complete.
I20260924 00:12:04.759137 14695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.759464 14695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.759656 14695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.759739 14695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:04.759796 14695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:04.759845 14695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:12:04.760851 14695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.760965 14695 leader_election.cc:302] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e346a7f6e5a4d73b5dc09a1f19ade82; no voters:
I20260924 00:12:04.761162 14695 leader_election.cc:288] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:12:04.761197 14699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:12:04.761315 14699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 LEADER]: Becoming Leader. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Running, Role: LEADER
I20260924 00:12:04.761406 14699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:04.761432 14695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:04.761775 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e346a7f6e5a4d73b5dc09a1f19ade82. Latest consensus state: current_term: 2 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 00:12:04.761816 14700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 00:12:04.761978 14700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:04.761929 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:04.762193 14707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:04.763006 14707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:04.763120 14707 catalog_manager.cc:1259] Loaded cluster ID: 3c94c7ef90dc47a58cfbd4fc87bedac3
I20260924 00:12:04.763162 14707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:12:04.763589 14707 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 00:12:04.768443 14707 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 00:12:04.770843 14707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:12:04.771040 14707 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Loaded TSK: 0
I20260924 00:12:04.771282 14707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:12:04.826924 14697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.827160 14697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.827200 14697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.828330 14697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.828392 14697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.828437 14697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:12:04.829928 14697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33289
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.830180 14697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.830401 14697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.836272 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.836447 14697 server_base.cc:1034] running on GCE node
W20260924 00:12:04.836593 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.836661 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.837692 14697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.837971 14697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.839131 14697 hybrid_clock.cc:648] HybridClock initialized: now 1790208724839122 us; error 27 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.848526 14697 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:12:04.849056 14697 webserver.cc:460] Webserver started at http://127.0.98.1:33289/ using document root <none> and password file <none>
I20260924 00:12:04.849287 14697 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.849344 14697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.850804 14697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:12:04.851526 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.851702 14697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:04.851797 14697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "c48da81c67ab4522823cb79da7239160"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "4b2e36e3cf2569355dbfdb4d93d20a7b"
server_key_iv: "937466148820d4a1ccdff6421e6468f1"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.852164 14697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:04.859766 14697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:04.862202 14697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:04.862354 14697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:04.862633 14697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:04.863131 14739 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:12:04.864563 14697 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:12:04.864615 14697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:12:04.864648 14697 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:12:04.865316 14697 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:12:04.865360 14697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:04.865404 14739 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Bootstrap starting.
I20260924 00:12:04.873198 14697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42003
I20260924 00:12:04.873734 14697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:12:04.874306 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42003 every 8 connection(s)
I20260924 00:12:04.877161 14739 log.cc:824] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:04.879199 14739 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:04.879544 14739 tablet_bootstrap.cc:492] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Bootstrap complete.
I20260924 00:12:04.880136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14697
I20260924 00:12:04.880425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43385
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=39483
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:04.880718 14739 ts_tablet_manager.cc:1397] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260924 00:12:04.882239 14739 raft_consensus.cc:348] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.882575 14739 raft_consensus.cc:729] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48da81c67ab4522823cb79da7239160, State: Initialized, Role: FOLLOWER
I20260924 00:12:04.882710 14739 consensus_queue.cc:260] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:04.882967 14739 ts_tablet_manager.cc:1428] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:04.888010 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.874868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37445 (local address 127.0.98.62:38687)
0924 00:12:04.875772 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.875780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.875817 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:12:04.875946 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.875951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.876004 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.876140 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.877121 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.878497 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.879956 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.880944 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.881532 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.883723 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.883727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.883728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.883740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.885837 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.886392 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.886395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.886395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.886524 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.886838 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.886839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.886840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.887179 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.887292 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.887688 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.887819 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":89,"threads_started":1}
I20260924 00:12:04.888052 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.874486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:37445)
0924 00:12:04.874842 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.874887 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.875570 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.876216 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.876222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.876675 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:04.876971 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.876979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.878612 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.878613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.879871 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.879874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.881083 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.881084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.881222 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.881844 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:04.881853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:04.883603 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:04.885931 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.885940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.885942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.886282 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.886597 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.886598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.886598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.886750 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.887284 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:04.887292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:04.887471 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.887602 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.887796 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":88,"threads_started":1}
I20260924 00:12:04.889103 14847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
I20260924 00:12:04.889220 14847 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:04.889420 14847 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:04.890290 14654 ts_manager.cc:194] Registered new tserver with Master: c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003)
I20260924 00:12:04.891491 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37445
I20260924 00:12:04.892557 14847 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:12:04.905011 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.890981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:43532)
0924 00:12:04.891770 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:04.891800 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:04.891876 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:04.892112 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:04.892115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:04.892436 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:04.892486 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.892488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.893903 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.893904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.894900 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:04.894903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.895558 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.895562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:04.895638 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.896178 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:04.896183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:04.901429 (+ 5246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:04.903647 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.903655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.903657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.904003 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.904234 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:04.904234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:04.904235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:04.904291 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:04.904631 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:04.904637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:04.904690 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:12:04.904765 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:12:04.904899 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":95,"threads_started":1}
I20260924 00:12:04.905815 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:04.891257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.0.98.1:42003)
0924 00:12:04.891400 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:04.891405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:04.891951 (+ 546us) server_negotiation.cc:407] Connection header received
0924 00:12:04.891986 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:04.891988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:04.892004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:04.892059 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:04.892566 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.893821 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.895004 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:04.895483 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:04.895552 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:04.901552 (+ 6000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:04.901561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:04.901561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:04.901569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:04.903551 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.904102 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.904106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.904106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.904168 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:04.904350 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:04.904350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:04.904351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:04.904522 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:04.904605 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:04.905613 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 00:12:04.905715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 00:12:04.906199 14763 tablet_service.cc:1514] Processing DeleteTablet for tablet 1709392914be47a69794a1ca5c410725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43532
I20260924 00:12:04.906723 14856 tablet_replica.cc:330] stopping tablet replica
I20260924 00:12:04.906836 14856 raft_consensus.cc:2229] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:04.906908 14856 raft_consensus.cc:2258] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:04.907193 14856 ts_tablet_manager.cc:1905] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:04.909159 14856 ts_tablet_manager.cc:1918] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:04.909222 14856 log.cc:1196] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/1709392914be47a69794a1ca5c410725
I20260924 00:12:04.909482 14856 ts_tablet_manager.cc:1939] T 1709392914be47a69794a1ca5c410725 P c48da81c67ab4522823cb79da7239160: Deleting consensus metadata
I20260924 00:12:04.909996 14643 catalog_manager.cc:4841] TS c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003): tablet 1709392914be47a69794a1ca5c410725 (table test-table [id=031386dbf2e64a2cbd08fdc7183f99e0]) successfully deleted
W20260924 00:12:04.970284 14852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:04.970516 14852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:04.970553 14852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:04.971772 14852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:04.971838 14852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:04.971884 14852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:12:04.973336 14852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39483
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:04.973596 14852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:04.973815 14852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:04.978785 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.978760 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:04.978799 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:04.978971 14852 server_base.cc:1034] running on GCE node
I20260924 00:12:04.979207 14852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:04.979491 14852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:04.980643 14852 hybrid_clock.cc:648] HybridClock initialized: now 1790208724980632 us; error 42 us; skew 500 ppm
Sep 24 00:12:04 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:04.982995 14852 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:12:04.983494 14852 webserver.cc:460] Webserver started at http://127.0.98.2:39483/ using document root <none> and password file <none>
I20260924 00:12:04.983721 14852 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:04.983799 14852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:04.985145 14852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:12:04.985711 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:04.985875 14852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:04.985970 14852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "739f976789ae4192b64960f1d247b7f1"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "ece49261838b64dc40329e0bc9e8978b"
server_key_iv: "c4484501bdf11ce8f3400e13e4f15cf3"
server_key_version: "encryptionkey@0"
I20260924 00:12:04.986320 14852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:05.005326 14852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:05.007319 14852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:05.007494 14852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:05.007748 14852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:05.008091 14852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:05.008144 14852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.008180 14852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:05.008198 14852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.014202 14852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43385
I20260924 00:12:05.014235 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43385 every 8 connection(s)
I20260924 00:12:05.014664 14852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:12:05.016987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14852
I20260924 00:12:05.017270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44261
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=39791
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:05.027179 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.015437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48051 (local address 127.0.98.62:38687)
0924 00:12:05.015502 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.015505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.015792 (+ 287us) server_negotiation.cc:407] Connection header received
0924 00:12:05.016627 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.016628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.016640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.016678 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:05.018224 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.019007 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.020648 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.021020 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.021103 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.024222 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:05.024223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:05.024224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:05.024231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:05.025617 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.026129 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.026130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.026130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.026179 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.026483 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.026483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.026484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.026585 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:05.026633 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.027037 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.027109 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:12:05.027566 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.015436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:48051)
0924 00:12:05.015614 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.015671 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.016485 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.016773 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.016778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.017819 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:05.018099 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.018107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.019137 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.019138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.020551 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.020554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.021114 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.021115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.021254 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.021983 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:05.021994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:05.024127 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:05.025698 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:05.025706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:05.025709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:05.026052 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:05.026243 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:05.026244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:05.026244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:05.026398 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:05.026649 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:05.026656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:05.026886 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.027023 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.027261 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260924 00:12:05.028090 14985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
I20260924 00:12:05.028204 14985 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:05.028431 14985 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:05.028842 14654 ts_manager.cc:194] Registered new tserver with Master: 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
I20260924 00:12:05.029212 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48051
W20260924 00:12:05.103983 14988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:05.104214 14988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:05.104254 14988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:05.105654 14988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:05.105705 14988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:05.105746 14988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:12:05.107343 14988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39791
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:05.107575 14988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:05.107812 14988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:05.112864 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.112852 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.112856 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.113075 14988 server_base.cc:1034] running on GCE node
I20260924 00:12:05.113298 14988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:05.113488 14988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:05.114650 14988 hybrid_clock.cc:648] HybridClock initialized: now 1790208725114639 us; error 35 us; skew 500 ppm
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:05.117223 14988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:12:05.117668 14988 webserver.cc:460] Webserver started at http://127.0.98.3:39791/ using document root <none> and password file <none>
I20260924 00:12:05.117882 14988 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:05.117957 14988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:05.119382 14988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:05.119923 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.120069 14988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.120162 14988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "3e49aa0f487147a89d477d8dfcec2e44"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "f346b8900aa4695a334c79fe822f3df4"
server_key_iv: "5b8c51fcca2474decee2c9ad8905245c"
server_key_version: "encryptionkey@0"
I20260924 00:12:05.120496 14988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:05.149406 14988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:05.151765 14988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:05.151930 14988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:05.152168 14988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:05.152491 14988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:05.152549 14988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.152591 14988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:05.152621 14988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.158500 14988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44261
I20260924 00:12:05.158555 15116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44261 every 8 connection(s)
I20260924 00:12:05.158885 14988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:12:05.163412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 14988
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:05.169627 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.159628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57077 (local address 127.0.98.62:38687)
0924 00:12:05.159695 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.159696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.159906 (+ 210us) server_negotiation.cc:407] Connection header received
0924 00:12:05.160437 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.160439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.160452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.160497 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:05.161478 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.162061 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.163256 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.163450 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.163535 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.166557 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:05.166559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:05.166559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:05.166567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:05.168106 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.168600 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.168601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.168601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.168662 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.168941 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.168942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.168942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.169052 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:05.169107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.169433 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.169526 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:12:05.169808 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.159611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:57077)
0924 00:12:05.159788 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.159824 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.160326 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.160591 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.160600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.161158 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:05.161348 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.161394 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.162157 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.162158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.163143 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.163146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.163551 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.163552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.163700 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.164882 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:05.164893 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:05.166469 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:05.168186 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:05.168193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:05.168195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:05.168514 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:05.168730 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:05.168730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:05.168731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:05.168867 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:05.169117 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:05.169125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:05.169291 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.169419 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.169585 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260924 00:12:05.170276 15117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38687
I20260924 00:12:05.170385 15117 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:05.170576 15117 heartbeater.cc:511] Master 127.0.98.62:38687 requested a full tablet report, sending...
I20260924 00:12:05.170794 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.163825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.0.98.62:38687)
0924 00:12:05.163952 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.163954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.163995 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:12:05.164068 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.164068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.164080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.164144 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:05.164669 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.165361 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.166096 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.166377 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.166456 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.167774 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:05.167775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:05.167776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:05.167781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:05.169425 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.170019 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.170020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.170021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.170074 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:05.170282 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:05.170283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:05.170283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:05.170408 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:05.170582 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.170618 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.170692 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260924 00:12:05.171059 14654 ts_manager.cc:194] Registered new tserver with Master: 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261)
I20260924 00:12:05.171486 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57077
I20260924 00:12:05.171494 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:12:05.173905 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.171820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.0.98.62:38687)
0924 00:12:05.171861 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.171863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.171948 (+ 85us) server_negotiation.cc:407] Connection header received
0924 00:12:05.172025 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.172026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.172053 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.172108 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:05.172224 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.172660 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.173325 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.173499 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.173556 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.173665 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:05.173744 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:12:05.173796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.173815 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.173828 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:12:05.174166 14654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:12:05.175824 14654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:12:05.177949 14763 tablet_service.cc:1467] Processing CreateTablet for tablet 15175b146b3e4360929b29af4f29e972 (DEFAULT_TABLE table=test-table [id=ac545982cdda4bbd946b827297e51050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:05.178161 14763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15175b146b3e4360929b29af4f29e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:05.180299 14739 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Bootstrap starting.
I20260924 00:12:05.180636 14739 tablet_bootstrap.cc:654] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:05.180985 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.177539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43385 (local address 127.0.0.1:56712)
0924 00:12:05.177674 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.177682 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.177723 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.177984 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.177985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.177988 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:05.178028 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.178030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.179111 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.179112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.179933 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.179936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.180724 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.180724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.180787 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.180805 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.180864 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.180901 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260924 00:12:05.180997 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.177654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.0.98.2:43385)
0924 00:12:05.177825 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.177829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.177838 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:05.177876 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.177878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.177889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.177975 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:05.178087 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.179003 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.180027 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.180636 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.180705 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.180764 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.180878 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.180913 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260924 00:12:05.181622 14739 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: No bootstrap required, opened a new log
I20260924 00:12:05.181659 14739 ts_tablet_manager.cc:1397] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:05.181773 14739 raft_consensus.cc:348] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.181819 14739 raft_consensus.cc:374] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:05.181846 14739 raft_consensus.cc:729] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48da81c67ab4522823cb79da7239160, State: Initialized, Role: FOLLOWER
I20260924 00:12:05.181737 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 15175b146b3e4360929b29af4f29e972 (DEFAULT_TABLE table=test-table [id=ac545982cdda4bbd946b827297e51050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:05.181903 14739 consensus_queue.cc:260] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.182006 14739 ts_tablet_manager.cc:1428] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.182090 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15175b146b3e4360929b29af4f29e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:05.182324 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.177486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44261 (local address 127.0.0.1:40398)
0924 00:12:05.177531 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.177541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.177576 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.178028 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.178030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.178031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:05.178073 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.178074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.179383 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.179384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.180215 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.180218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.182114 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.182115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.182186 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.182191 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.182222 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.182264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 00:12:05.182689 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.177646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.98.3:44261)
0924 00:12:05.177858 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.177862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.177871 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:05.177906 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.177907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.177922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.178177 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:05.178238 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.179260 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.181323 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.182007 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.182481 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.182554 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.182580 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.182619 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260924 00:12:05.183203 15051 tablet_service.cc:1467] Processing CreateTablet for tablet 15175b146b3e4360929b29af4f29e972 (DEFAULT_TABLE table=test-table [id=ac545982cdda4bbd946b827297e51050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:05.183492 15051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15175b146b3e4360929b29af4f29e972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:05.185102 15127 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Bootstrap starting.
I20260924 00:12:05.185655 15127 tablet_bootstrap.cc:654] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:05.186010 15128 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Bootstrap starting.
I20260924 00:12:05.186240 15127 log.cc:824] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:05.186725 15128 tablet_bootstrap.cc:654] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:05.187074 15127 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: No bootstrap required, opened a new log
I20260924 00:12:05.187162 15127 ts_tablet_manager.cc:1397] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:12:05.187341 15128 log.cc:824] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:05.188064 15128 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: No bootstrap required, opened a new log
I20260924 00:12:05.188138 15128 ts_tablet_manager.cc:1397] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:12:05.189253 15127 raft_consensus.cc:348] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.189441 15127 raft_consensus.cc:374] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:05.189486 15127 raft_consensus.cc:729] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739f976789ae4192b64960f1d247b7f1, State: Initialized, Role: FOLLOWER
I20260924 00:12:05.189615 15127 consensus_queue.cc:260] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.189891 15127 ts_tablet_manager.cc:1428] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:12:05.190026 14985 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
I20260924 00:12:05.190138 15128 raft_consensus.cc:348] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.190303 15128 raft_consensus.cc:374] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:05.190344 15128 raft_consensus.cc:729] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e49aa0f487147a89d477d8dfcec2e44, State: Initialized, Role: FOLLOWER
I20260924 00:12:05.190456 15128 consensus_queue.cc:260] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.190729 15128 ts_tablet_manager.cc:1428] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:12:05.190752 15117 heartbeater.cc:503] Master 127.0.98.62:38687 was elected leader, sending a full tablet report...
W20260924 00:12:05.265296 14986 tablet.cc:2369] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:12:05.374935 14848 tablet.cc:2369] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:12:05.409474 15118 tablet.cc:2369] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:12:05.513525 15132 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:05.513666 15132 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.514014 15132 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385), c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003)
I20260924 00:12:05.516961 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.514231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43385 (local address 127.0.98.3:47935)
0924 00:12:05.514300 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.514315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.514386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.514550 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.514551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.514553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:05.514635 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.514637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.515332 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.515333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.516106 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.516109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.516682 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.516683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.516769 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.516774 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.516838 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.516882 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:12:05.516980 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.514307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47935 (local address 127.0.98.2:43385)
0924 00:12:05.514380 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.514383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.514396 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:12:05.514435 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.514436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.514449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.514512 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:05.514699 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.515179 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.516201 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.516590 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.516679 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.516690 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.516880 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.516929 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:12:05.517340 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15175b146b3e4360929b29af4f29e972" candidate_uuid: "3e49aa0f487147a89d477d8dfcec2e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f976789ae4192b64960f1d247b7f1" is_pre_election: true
I20260924 00:12:05.517522 14939 raft_consensus.cc:2454] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e49aa0f487147a89d477d8dfcec2e44 in term 0.
I20260924 00:12:05.517810 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.514562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51051 (local address 127.0.98.1:42003)
0924 00:12:05.514690 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.514692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.514699 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:12:05.514721 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.514721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.514744 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.514826 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:05.514995 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.515783 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.516855 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.517405 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.517463 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.517555 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.517706 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.517758 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260924 00:12:05.517768 15004 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1; no voters:
I20260924 00:12:05.517845 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.514358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.98.3:51051)
0924 00:12:05.514550 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.514561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.514609 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.514845 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:05.514846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:05.514848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:05.514925 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.514926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.515886 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.515887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.516626 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:05.516628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.517516 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.517517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:05.517632 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.517637 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:12:05.517698 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 00:12:05.517734 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260924 00:12:05.518092 15132 raft_consensus.cc:2781] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:12:05.518174 15132 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:12:05.518205 15132 raft_consensus.cc:3037] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:05.518174 14790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15175b146b3e4360929b29af4f29e972" candidate_uuid: "3e49aa0f487147a89d477d8dfcec2e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48da81c67ab4522823cb79da7239160" is_pre_election: true
I20260924 00:12:05.518333 14790 raft_consensus.cc:2454] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e49aa0f487147a89d477d8dfcec2e44 in term 0.
I20260924 00:12:05.518956 15132 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.519105 15132 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [CANDIDATE]: Term 1 election: Requested vote from peers 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385), c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003)
I20260924 00:12:05.519318 14790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15175b146b3e4360929b29af4f29e972" candidate_uuid: "3e49aa0f487147a89d477d8dfcec2e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48da81c67ab4522823cb79da7239160"
I20260924 00:12:05.519315 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15175b146b3e4360929b29af4f29e972" candidate_uuid: "3e49aa0f487147a89d477d8dfcec2e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739f976789ae4192b64960f1d247b7f1"
I20260924 00:12:05.519390 14790 raft_consensus.cc:3037] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:05.519410 14939 raft_consensus.cc:3037] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:05.520141 14790 raft_consensus.cc:2454] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e49aa0f487147a89d477d8dfcec2e44 in term 1.
I20260924 00:12:05.520293 14939 raft_consensus.cc:2454] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e49aa0f487147a89d477d8dfcec2e44 in term 1.
I20260924 00:12:05.520294 15004 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e49aa0f487147a89d477d8dfcec2e44, c48da81c67ab4522823cb79da7239160; no voters:
I20260924 00:12:05.520388 15132 raft_consensus.cc:2781] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:05.520545 15132 raft_consensus.cc:686] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 LEADER]: Becoming Leader. State: Replica: 3e49aa0f487147a89d477d8dfcec2e44, State: Running, Role: LEADER
I20260924 00:12:05.520655 15132 consensus_queue.cc:237] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.521467 14659 catalog_manager.cc:5495] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3e49aa0f487147a89d477d8dfcec2e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } health_report { overall_health: UNKNOWN } } }
I20260924 00:12:05.533077 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.530723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.0.98.3:44261)
0924 00:12:05.530773 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:05.530775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:05.530785 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:12:05.530843 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:05.530844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:05.530867 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:05.530955 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:05.531062 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.531599 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.532308 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:05.532500 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:05.532598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:05.532749 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:05.532848 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:12:05.532929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:05.532960 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 00:12:05.532980 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:12:05.534943 14790 raft_consensus.cc:1261] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Refusing update from remote peer 3e49aa0f487147a89d477d8dfcec2e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:05.534977 14939 raft_consensus.cc:1261] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Refusing update from remote peer 3e49aa0f487147a89d477d8dfcec2e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:05.535359 15135 consensus_queue.cc:1035] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:05.535487 15132 consensus_queue.cc:1035] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:05.538334 15140 mvcc.cc:205] Tried to move back new op lower bound from 7332694939788214272 to 7332694939733168128. Current Snapshot: MvccSnapshot[applied={T|T < 7332694939788214272 or (T in {7332694939788214272})}]
I20260924 00:12:05.539935 14654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:05.540017 14654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:05.540741 14654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Sending DeleteTablet for 3 replicas of tablet 15175b146b3e4360929b29af4f29e972
I20260924 00:12:05.541072 14763 tablet_service.cc:1514] Processing DeleteTablet for tablet 15175b146b3e4360929b29af4f29e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43532
I20260924 00:12:05.541074 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 15175b146b3e4360929b29af4f29e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56712
I20260924 00:12:05.541180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14697
I20260924 00:12:05.541424 15148 tablet_replica.cc:330] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: stopping tablet replica
I20260924 00:12:05.541553 15148 raft_consensus.cc:2229] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:05.541651 15148 raft_consensus.cc:2258] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:05.542093 15148 ts_tablet_manager.cc:1905] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:05.542080 15051 tablet_service.cc:1514] Processing DeleteTablet for tablet 15175b146b3e4360929b29af4f29e972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40398
I20260924 00:12:05.544145 15148 ts_tablet_manager.cc:1918] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:05.544260 15148 log.cc:1196] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/15175b146b3e4360929b29af4f29e972
I20260924 00:12:05.544565 15148 ts_tablet_manager.cc:1939] T 15175b146b3e4360929b29af4f29e972 P 739f976789ae4192b64960f1d247b7f1: Deleting consensus metadata
I20260924 00:12:05.545014 14643 catalog_manager.cc:4841] TS 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): tablet 15175b146b3e4360929b29af4f29e972 (table test-table [id=ac545982cdda4bbd946b827297e51050]) successfully deleted
W20260924 00:12:05.548039 14643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42003 (error 108)
I20260924 00:12:05.548425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14852
I20260924 00:12:05.548519 15149 tablet_replica.cc:330] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: stopping tablet replica
I20260924 00:12:05.548563 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.548412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42003 (local address 127.0.0.1:43538)
0924 00:12:05.548477 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.548523 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 00:12:05.548664 14643 catalog_manager.cc:4584] TS c48da81c67ab4522823cb79da7239160 (127.0.98.1:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15175b146b3e4360929b29af4f29e972: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42003: connect: Connection refused (error 111)
I20260924 00:12:05.549139 15149 raft_consensus.cc:2229] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:12:05.549263 15149 raft_consensus.cc:2258] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:05.550155 15149 ts_tablet_manager.cc:1905] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:05.552201 15149 ts_tablet_manager.cc:1918] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:05.552332 15149 log.cc:1196] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/15175b146b3e4360929b29af4f29e972
I20260924 00:12:05.552680 15149 ts_tablet_manager.cc:1939] T 15175b146b3e4360929b29af4f29e972 P 3e49aa0f487147a89d477d8dfcec2e44: Deleting consensus metadata
I20260924 00:12:05.553174 14644 catalog_manager.cc:4841] TS 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): tablet 15175b146b3e4360929b29af4f29e972 (table test-table [id=ac545982cdda4bbd946b827297e51050]) successfully deleted
I20260924 00:12:05.554015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14988
I20260924 00:12:05.558561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 14626
I20260924 00:12:05.563722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--webserver_interface=127.0.98.62
--webserver_port=39811
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:05.644716 15151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:05.644924 15151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:05.644946 15151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:05.646030 15151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:05.646072 15151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:05.646086 15151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:05.646099 15151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:05.646111 15151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:05.647639 15151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:05.647874 15151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:05.648063 15151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:05.652825 15157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.652948 15151 server_base.cc:1034] running on GCE node
W20260924 00:12:05.652837 15156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.653100 15159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.653353 15151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:05.653638 15151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:05.654779 15151 hybrid_clock.cc:648] HybridClock initialized: now 1790208725654767 us; error 29 us; skew 500 ppm
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:05.657186 15151 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:05.657639 15151 webserver.cc:460] Webserver started at http://127.0.98.62:39811/ using document root <none> and password file <none>
I20260924 00:12:05.657816 15151 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:05.657856 15151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:05.659152 15151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:05.659812 15166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.659972 15151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.660079 15151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82"
format_stamp: "Formatted at 2026-09-24 00:12:03 on dist-test-slave-3q8k"
server_key: "0bb04e6c42076c5cd16f2755f803a939"
server_key_iv: "2decabbd816f6657f10bb495c1fe1e75"
server_key_version: "encryptionkey@0"
I20260924 00:12:05.660429 15151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:05.679751 15151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:05.681615 15151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:05.681793 15151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:05.685952 15151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38687
I20260924 00:12:05.686005 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38687 every 8 connection(s)
I20260924 00:12:05.686389 15151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:05.687956 15219 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:12:05.688445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15151
I20260924 00:12:05.688678 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap starting.
I20260924 00:12:05.688752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42003
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=33289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:05.696945 15219 log.cc:824] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:05.699148 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:05.699481 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82: Bootstrap complete.
I20260924 00:12:05.701776 15219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:05.702014 15219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Initialized, Role: FOLLOWER
I20260924 00:12:05.702212 15219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:05.702301 15219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:05.702353 15219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:05.702401 15219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:12:05.703418 15219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:05.703531 15219 leader_election.cc:302] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e346a7f6e5a4d73b5dc09a1f19ade82; no voters:
I20260924 00:12:05.703719 15219 leader_election.cc:288] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:12:05.703758 15223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:12:05.703873 15223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [term 3 LEADER]: Becoming Leader. State: Replica: 8e346a7f6e5a4d73b5dc09a1f19ade82, State: Running, Role: LEADER
I20260924 00:12:05.703956 15223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 00:12:05.703985 15219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:05.704624 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e346a7f6e5a4d73b5dc09a1f19ade82. Latest consensus state: current_term: 3 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 00:12:05.704612 15224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e346a7f6e5a4d73b5dc09a1f19ade82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 00:12:05.704711 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:05.704718 15224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e346a7f6e5a4d73b5dc09a1f19ade82 [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:05.704896 15232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:05.705688 15232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:05.705786 15232 catalog_manager.cc:1259] Loaded cluster ID: 3c94c7ef90dc47a58cfbd4fc87bedac3
I20260924 00:12:05.705811 15232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 00:12:05.706004 15232 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1a984899ad google::LogMessage::Fail() at ??:0
@ 0x7f1a9848c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1a984894de google::LogMessage::Flush() at ??:0
@ 0x7f1a9848aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1aa03ae0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1aa03b78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1a9a3570d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1a9a34c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1a9b1b06db start_thread at ??:0
@ 0x7f1a96f8761f clone at ??:0
W20260924 00:12:05.777573 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:05.777793 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:05.777829 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:05.778954 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:05.779016 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:05.779062 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:12:05.780520 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33289
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:05.780783 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:05.781006 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:05.785943 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.785959 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.786134 15221 server_base.cc:1034] running on GCE node
W20260924 00:12:05.785945 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.786419 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:05.786672 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:05.787973 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790208725787954 us; error 32 us; skew 500 ppm
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:05.790438 15221 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:12:05.790934 15221 webserver.cc:460] Webserver started at http://127.0.98.1:33289/ using document root <none> and password file <none>
I20260924 00:12:05.791169 15221 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:05.791249 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:05.792593 15221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:05.793287 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.793504 15221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:05.793602 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "c48da81c67ab4522823cb79da7239160"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "4b2e36e3cf2569355dbfdb4d93d20a7b"
server_key_iv: "937466148820d4a1ccdff6421e6468f1"
server_key_version: "encryptionkey@0"
I20260924 00:12:05.794044 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:05.814566 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:05.817013 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:05.817127 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:05.817351 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:05.817881 15272 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:12:05.819382 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:12:05.819432 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:12:05.819473 15221 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:12:05.820447 15221 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:12:05.820490 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:05.820581 15272 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Bootstrap starting.
I20260924 00:12:05.827857 15221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42003
I20260924 00:12:05.827939 15379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42003 every 8 connection(s)
I20260924 00:12:05.828302 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:12:05.830718 15272 log.cc:824] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:05.832864 15272 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:05.833158 15272 tablet_bootstrap.cc:492] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Bootstrap complete.
I20260924 00:12:05.833786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15221
I20260924 00:12:05.834064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43385
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=39483
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:05.834281 15272 ts_tablet_manager.cc:1397] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260924 00:12:05.835860 15272 raft_consensus.cc:348] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.836113 15272 raft_consensus.cc:729] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48da81c67ab4522823cb79da7239160, State: Initialized, Role: FOLLOWER
I20260924 00:12:05.836280 15272 consensus_queue.cc:260] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:05.836583 15272 ts_tablet_manager.cc:1428] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260924 00:12:05.911173 15382 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:12:05.829026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:46575)
0924 00:12:05.829256 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.829319 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:05.830118 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:05.910963 (+ 80845us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":109,"threads_started":1}
W20260924 00:12:05.911435 15380 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38687: Client connection negotiation failed: client connection to 127.0.98.62:38687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:12:05.911787 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.911658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:37779)
0924 00:12:05.911714 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.911759 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:12:05.912094 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.912002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:51339)
0924 00:12:05.912046 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.912073 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 00:12:05.912209 15380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:12:05.917052 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:05.917236 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:05.917260 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:05.918442 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:05.918486 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:05.918512 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:12:05.919967 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39483
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:05.920166 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:05.920339 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:05.925376 15391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.925381 15390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:05.925387 15393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:05.925726 15384 server_base.cc:1034] running on GCE node
I20260924 00:12:05.925894 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:05.926090 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:05.927253 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790208725927238 us; error 28 us; skew 500 ppm
Sep 24 00:12:05 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:05.929623 15384 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:12:05.930052 15384 webserver.cc:460] Webserver started at http://127.0.98.2:39483/ using document root <none> and password file <none>
I20260924 00:12:05.930249 15384 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:05.930291 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:05.931629 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:05.932164 15400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.932333 15384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:05.932415 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "739f976789ae4192b64960f1d247b7f1"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "ece49261838b64dc40329e0bc9e8978b"
server_key_iv: "c4484501bdf11ce8f3400e13e4f15cf3"
server_key_version: "encryptionkey@0"
I20260924 00:12:05.932740 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:05.948580 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:05.950371 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:05.950480 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:05.950659 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:05.950953 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:05.950985 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.951007 15384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:05.951041 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:05.956984 15384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43385
I20260924 00:12:05.957068 15513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43385 every 8 connection(s)
I20260924 00:12:05.957393 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:12:05.958415 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.958015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:58281)
0924 00:12:05.958168 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.958268 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260924 00:12:05.958792 15514 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38687: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
I20260924 00:12:05.959110 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.959021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:34203)
0924 00:12:05.959068 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.959094 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:12:05.959384 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:05.959331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:55157)
0924 00:12:05.959353 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:05.959371 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 00:12:05.959509 15514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:12:05.959617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15384
I20260924 00:12:05.959903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44261
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38687
--webserver_port=39791
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:06.040413 15517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:06.040627 15517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:06.040650 15517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:06.041801 15517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:06.041846 15517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:06.041895 15517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:12:06.043363 15517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39791
--tserver_master_addrs=127.0.98.62:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:06.043597 15517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:06.043823 15517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:06.049254 15523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:06.049302 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:06.049356 15517 server_base.cc:1034] running on GCE node
W20260924 00:12:06.049487 15522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:06.049693 15517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:06.049953 15517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:06.051110 15517 hybrid_clock.cc:648] HybridClock initialized: now 1790208726051097 us; error 32 us; skew 500 ppm
Sep 24 00:12:06 dist-test-slave-3q8k krb5kdc[14082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:06.053587 15517 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:12:06.054008 15517 webserver.cc:460] Webserver started at http://127.0.98.3:39791/ using document root <none> and password file <none>
I20260924 00:12:06.054224 15517 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:06.054291 15517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:06.055612 15517 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:06.056169 15532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:06.056324 15517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:06.056418 15517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "3e49aa0f487147a89d477d8dfcec2e44"
format_stamp: "Formatted at 2026-09-24 00:12:04 on dist-test-slave-3q8k"
server_key: "f346b8900aa4695a334c79fe822f3df4"
server_key_iv: "5b8c51fcca2474decee2c9ad8905245c"
server_key_version: "encryptionkey@0"
I20260924 00:12:06.056758 15517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:06.081136 15517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:06.083583 15517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:06.083742 15517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:06.083974 15517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:06.084283 15517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:06.084335 15517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:06.084378 15517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:06.084404 15517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:06.090082 15517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44261
I20260924 00:12:06.090188 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44261 every 8 connection(s)
I20260924 00:12:06.090498 15517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:12:06.091552 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:06.091155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:50203)
0924 00:12:06.091340 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:06.091410 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
W20260924 00:12:06.091957 15646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38687 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38687: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
I20260924 00:12:06.092231 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:06.092143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:32917)
0924 00:12:06.092183 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:06.092211 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 00:12:06.092515 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:06.092441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:52853)
0924 00:12:06.092479 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:06.092501 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 00:12:06.092646 15646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:12:06.094496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15517
I20260924 00:12:06.912755 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:06.912535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:41603)
0924 00:12:06.912655 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:06.912691 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 00:12:06.960121 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:06.959909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:42375)
0924 00:12:06.960036 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:06.960071 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260924 00:12:07.093165 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.092920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:52211)
0924 00:12:07.093084 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:07.093119 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260924 00:12:07.094727 15653 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:07.094839 15653 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:07.095211 15653 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
Sep 24 00:12:07 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 00:12:07 dist-test-slave-3q8k krb5kdc[14082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 00:12:07.107530 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.095568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37649 (local address 127.0.98.2:43385)
0924 00:12:07.095712 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:07.095719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:07.095748 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:12:07.096330 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:07.096334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:07.096394 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:07.096566 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:07.097380 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.098314 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.099624 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.100030 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.100169 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:07.102982 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:07.102986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:07.102986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:07.102997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:07.105305 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:07.105847 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:07.105853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:07.105853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:07.106009 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:07.106354 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:07.106356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:07.106356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:07.106707 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:07.106799 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:07.107210 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 00:12:07.107389 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260924 00:12:07.107527 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.095414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43385 (local address 127.0.98.1:37649)
0924 00:12:07.095470 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:07.095483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:07.095552 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:07.096621 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:07.096624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:07.097035 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:07.097226 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:07.097232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.098419 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.098420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:07.099510 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:07.099513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.100126 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.100128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:07.100252 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:07.100891 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:07.100900 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:07.102854 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:07.105398 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:07.105408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:07.105412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:07.105738 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:07.106106 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:07.106107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:07.106108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:07.106254 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:07.106776 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:07.106784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:07.106995 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 00:12:07.107134 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 00:12:07.107393 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":30}
I20260924 00:12:07.107631 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.095414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44261 (local address 127.0.98.1:57091)
0924 00:12:07.095686 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:07.095695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:07.095754 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:07.096688 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:07.096689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:07.097000 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:07.097225 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:07.097232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.098645 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.098646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:07.099508 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:07.099511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.100186 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.100187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:07.100253 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:07.100940 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:07.100945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:07.102493 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:07.105885 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:07.105886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:07.105886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:07.106131 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:07.106444 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:07.106445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:07.106446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:07.106519 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:07.107046 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:07.107047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:07.107136 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 00:12:07.107248 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:12:07.107479 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":156,"thread_start_us":60,"threads_started":1}
I20260924 00:12:07.107766 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.095534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57091 (local address 127.0.98.3:44261)
0924 00:12:07.095674 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:07.095686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:07.095744 (+ 58us) server_negotiation.cc:407] Connection header received
0924 00:12:07.096330 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:07.096334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:07.096394 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:07.096557 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:07.097398 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.098509 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.099600 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:07.100050 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:07.100187 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:07.102635 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:07.102640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:07.102641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:07.102654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:07.105288 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:07.106219 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:07.106223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:07.106223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:07.106357 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:07.106583 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:07.106585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:07.106585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:07.106959 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:07.107055 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:07.107338 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 00:12:07.107483 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260924 00:12:07.109051 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:07.109226 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:07.109277 15266 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:07.109385 15653 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:07.913534 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.913293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:59963)
0924 00:12:07.913445 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:07.913488 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260924 00:12:07.960716 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:07.960537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:54891)
0924 00:12:07.960644 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:07.960679 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260924 00:12:08.093852 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:08.093636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:36099)
0924 00:12:08.093771 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:08.093813 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260924 00:12:08.913189 15660 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:08.913280 15660 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:08.913488 15660 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
W20260924 00:12:08.913954 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:08.914091 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:08.914134 15266 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:08.914201 15660 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:08.914315 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:08.914010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:54201)
0924 00:12:08.914243 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:08.914279 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":45,"threads_started":1}
I20260924 00:12:08.961328 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:08.961142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:45047)
0924 00:12:08.961253 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:08.961290 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 00:12:09.094482 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:09.094297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:56489)
0924 00:12:09.094412 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:09.094443 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 00:12:09.915205 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:09.914895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:43323)
0924 00:12:09.915066 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:09.915145 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260924 00:12:09.962028 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:09.961805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:51193)
0924 00:12:09.961931 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:09.961973 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 00:12:10.095086 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:10.094891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:37479)
0924 00:12:10.095009 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:10.095044 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 00:12:10.700313 15667 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:10.700469 15667 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:10.700657 15667 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
W20260924 00:12:10.701058 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:10.701140 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:10.701170 15266 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:10.701236 15667 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:10.916078 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:10.915807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:50723)
0924 00:12:10.915989 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:10.916032 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260924 00:12:10.962682 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:10.962498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:43883)
0924 00:12:10.962611 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:10.962644 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 00:12:11.095975 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:11.095729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:54423)
0924 00:12:11.095864 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:11.095920 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260924 00:12:11.916878 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:11.916651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:42905)
0924 00:12:11.916780 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:11.916819 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260924 00:12:11.963367 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:11.963120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:33929)
0924 00:12:11.963282 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:11.963321 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
I20260924 00:12:12.096714 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:12.096454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:36259)
0924 00:12:12.096635 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:12.096673 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260924 00:12:12.917696 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:12.917459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:46179)
0924 00:12:12.917605 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:12.917648 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 00:12:12.964037 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:12.963807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:50431)
0924 00:12:12.963937 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:12.963983 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260924 00:12:13.097494 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:13.097256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:45861)
0924 00:12:13.097409 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:13.097451 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260924 00:12:13.918475 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:13.918226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:46967)
0924 00:12:13.918376 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:13.918420 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260924 00:12:13.964705 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:13.964468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:34545)
0924 00:12:13.964614 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:13.964654 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260924 00:12:14.098239 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:14.097959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:39675)
0924 00:12:14.098152 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:14.098195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":122,"threads_started":1}
I20260924 00:12:14.919274 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:14.918996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:39643)
0924 00:12:14.919137 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:14.919181 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260924 00:12:14.965641 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:14.965424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:58443)
0924 00:12:14.965556 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:14.965592 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260924 00:12:15.098956 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:15.098703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:41255)
0924 00:12:15.098848 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:15.098901 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":101,"threads_started":1}
I20260924 00:12:15.566499 15683 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:15.566628 15683 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:15.566864 15683 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
W20260924 00:12:15.567358 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:15.567456 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:15.567508 15267 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:15.567600 15683 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:15.920037 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:15.919775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:40535)
0924 00:12:15.919942 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:15.919987 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260924 00:12:15.966357 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:15.966102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:41891)
0924 00:12:15.966249 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:15.966288 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260924 00:12:16.099661 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:16.099450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:45111)
0924 00:12:16.099577 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:16.099618 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260924 00:12:16.920830 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:16.920591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:36611)
0924 00:12:16.920749 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:16.920783 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 00:12:16.967288 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:16.966995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:49147)
0924 00:12:16.967164 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:16.967194 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260924 00:12:17.100453 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:17.100231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:45787)
0924 00:12:17.100365 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:17.100407 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260924 00:12:17.491885 15690 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:17.491981 15690 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:17.492187 15690 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
W20260924 00:12:17.492683 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:17.492769 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:17.492834 15267 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:17.492919 15690 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:17.921609 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:17.921366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:51779)
0924 00:12:17.921510 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:17.921555 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260924 00:12:17.967913 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:17.967703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:52915)
0924 00:12:17.967834 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:17.967869 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 00:12:18.101209 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:18.100962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:41143)
0924 00:12:18.101122 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:18.101164 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260924 00:12:18.922346 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:18.922124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:33895)
0924 00:12:18.922260 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:18.922301 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260924 00:12:18.968677 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:18.968426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:35141)
0924 00:12:18.968592 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:18.968632 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260924 00:12:19.101959 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:19.101724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:44465)
0924 00:12:19.101868 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:19.101914 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260924 00:12:19.234872 15697 raft_consensus.cc:482] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:19.235122 15697 raft_consensus.cc:504] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e49aa0f487147a89d477d8dfcec2e44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44261 } } peers { permanent_uuid: "739f976789ae4192b64960f1d247b7f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43385 } } peers { permanent_uuid: "c48da81c67ab4522823cb79da7239160" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42003 } }
I20260924 00:12:19.235385 15697 leader_election.cc:288] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261), 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385)
W20260924 00:12:19.235770 15266 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 739f976789ae4192b64960f1d247b7f1 (127.0.98.2:43385): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
W20260924 00:12:19.235857 15267 leader_election.cc:341] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e49aa0f487147a89d477d8dfcec2e44 (127.0.98.3:44261): Not found: Tablet not found: 15175b146b3e4360929b29af4f29e972
I20260924 00:12:19.235922 15267 leader_election.cc:302] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c48da81c67ab4522823cb79da7239160; no voters: 3e49aa0f487147a89d477d8dfcec2e44, 739f976789ae4192b64960f1d247b7f1
I20260924 00:12:19.236012 15697 raft_consensus.cc:2726] T 15175b146b3e4360929b29af4f29e972 P c48da81c67ab4522823cb79da7239160 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:12:19.923075 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:19.922855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:42719)
0924 00:12:19.922983 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:19.923025 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 00:12:19.969396 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:19.969195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:49527)
0924 00:12:19.969317 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:19.969349 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 00:12:20.102663 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:20.102452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:59387)
0924 00:12:20.102577 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:20.102619 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260924 00:12:20.923857 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:20.923629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.1:55555)
0924 00:12:20.923752 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:20.923785 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 00:12:20.970242 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:20.969977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.2:42267)
0924 00:12:20.970149 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:20.970195 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
I20260924 00:12:21.103561 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:21.103359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38687 (local address 127.0.98.3:34847)
0924 00:12:21.103482 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:21.103520 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260924 00:12:21.104573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15221
I20260924 00:12:21.109414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15384
I20260924 00:12:21.113670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15517
2026-09-24T00:12:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18432 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:12:21 dist-test-slave-3q8k krb5kdc[15706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:12:21 dist-test-slave-3q8k krb5kdc[15706](info): set up 2 sockets
Sep 24 00:12:21 dist-test-slave-3q8k krb5kdc[15706](info): commencing operation
krb5kdc: starting...
W20260924 00:12:22.155532 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:12:22.274823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:22.354863 15719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:22.355190 15719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:22.355243 15719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:22.356454 15719 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:12:22.356531 15719 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:12:22.356552 15719 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:12:22.356570 15719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:22.356599 15719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:22.356629 15719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:22.356658 15719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:22.356690 15719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:22.358209 15719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:22.358496 15719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:22.358733 15719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:22.363936 15724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.363979 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.364030 15719 server_base.cc:1034] running on GCE node
W20260924 00:12:22.363953 15725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.364410 15719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:22.364717 15719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:22.365883 15719 hybrid_clock.cc:648] HybridClock initialized: now 1790208742365874 us; error 55 us; skew 500 ppm
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:22.368505 15719 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:22.369004 15719 webserver.cc:460] Webserver started at http://127.0.98.62:36023/ using document root <none> and password file <none>
I20260924 00:12:22.369261 15719 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:22.369330 15719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:22.369485 15719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:22.370604 15719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "7dfefa502e81463484c132f33fb7608e"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "b81dd85a8329b6b2e62a18386f852137"
server_key_iv: "100d919084a5d0dc2a97bde7498cc2d0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.370988 15719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dfefa502e81463484c132f33fb7608e"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "b81dd85a8329b6b2e62a18386f852137"
server_key_iv: "100d919084a5d0dc2a97bde7498cc2d0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.372430 15719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:12:22.373204 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.373389 15719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:22.373495 15719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7dfefa502e81463484c132f33fb7608e"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "b81dd85a8329b6b2e62a18386f852137"
server_key_iv: "100d919084a5d0dc2a97bde7498cc2d0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.373574 15719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:22.390991 15719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:22.393567 15719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:22.393739 15719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:22.397991 15719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42949
I20260924 00:12:22.398046 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42949 every 8 connection(s)
I20260924 00:12:22.398424 15719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:22.399370 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:22.399745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15719
I20260924 00:12:22.399849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:12:22.400055 392 external_mini_cluster.cc:1428] Setting key 9237f270a9039c98cc00321245af0b1d
I20260924 00:12:22.402076 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap starting.
I20260924 00:12:22.402693 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:22.402987 15787 log.cc:824] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:22.403678 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: No bootstrap required, opened a new log
I20260924 00:12:22.405436 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:22.405587 15787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:22.405637 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Initialized, Role: FOLLOWER
I20260924 00:12:22.405789 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:22.405889 15787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:22.405934 15787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:22.405980 15787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:22.406579 15787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:22.406682 15787 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfefa502e81463484c132f33fb7608e; no voters:
I20260924 00:12:22.406890 15787 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:12:22.406962 15792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:22.407135 15792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 LEADER]: Becoming Leader. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Running, Role: LEADER
I20260924 00:12:22.407267 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:22.408031 15792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:22.411060 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfefa502e81463484c132f33fb7608e. Latest consensus state: current_term: 1 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:12:22.411144 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:22.411371 15799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:22.411974 15799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:22.412328 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:12:22.412406 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:22.414505 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.400564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.62:42949)
0924 00:12:22.400753 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.400764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.400796 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:12:22.401286 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.401292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.401341 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.401542 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:22.402187 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.403477 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.404332 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.404757 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.404882 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.407943 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:22.407948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:22.407948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:22.407961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:22.412743 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.413195 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.413198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.413198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.413295 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.413543 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.413543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.413544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.413803 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:22.414033 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.414110 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.414273 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260924 00:12:22.423663 15799 catalog_manager.cc:1347] Generated new cluster ID: 7427c371d4004dfbb08209071f575b3c
I20260924 00:12:22.423725 15799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:12:22.444275 15799 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:12:22.444825 15799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:12:22.449685 15799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Generated new TSK 0
I20260924 00:12:22.449868 15799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:12:22.478386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:22.560622 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:22.560864 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:22.560905 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:22.562111 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:22.562178 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:22.562223 15813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:12:22.563822 15813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:22.564086 15813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:22.564316 15813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:22.570169 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.570187 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.570335 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.570587 15813 server_base.cc:1034] running on GCE node
I20260924 00:12:22.570785 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:22.571033 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:22.572204 15813 hybrid_clock.cc:648] HybridClock initialized: now 1790208742572172 us; error 51 us; skew 500 ppm
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:22.574793 15813 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:12:22.575341 15813 webserver.cc:460] Webserver started at http://127.0.98.1:45581/ using document root <none> and password file <none>
I20260924 00:12:22.575590 15813 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:22.575635 15813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:22.575779 15813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:22.577023 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "e5339ebab4416d95cbbcc92c26bea56c"
server_key_iv: "d1a37261f606f7f98c27df5f8a543ab0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.577432 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "e5339ebab4416d95cbbcc92c26bea56c"
server_key_iv: "d1a37261f606f7f98c27df5f8a543ab0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.578809 15813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:12:22.579681 15828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.579878 15813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:22.579945 15813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "e5339ebab4416d95cbbcc92c26bea56c"
server_key_iv: "d1a37261f606f7f98c27df5f8a543ab0"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.580039 15813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:22.588052 15813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:22.590415 15813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:22.590564 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:22.590798 15813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:22.591121 15813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:22.591177 15813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.591238 15813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:22.591269 15813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.598604 15813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44507
I20260924 00:12:22.598694 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44507 every 8 connection(s)
I20260924 00:12:22.599042 15813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:12:22.603341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15813
I20260924 00:12:22.603456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:12:22.603677 392 external_mini_cluster.cc:1428] Setting key cf19b4909e6b47bfe196e3060c948f46
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:22.610150 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.599868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54529 (local address 127.0.98.62:42949)
0924 00:12:22.599926 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.599928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.600124 (+ 196us) server_negotiation.cc:407] Connection header received
0924 00:12:22.600620 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.600621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.600634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.600711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:22.601654 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.602422 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.603595 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.603899 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.603965 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.606558 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:22.606560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:22.606560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:22.606569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:22.608351 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.608925 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.608927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.608928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.608993 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.609349 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.609350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.609350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.609496 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:22.609573 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.609959 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.610055 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:12:22.610425 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.599838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:54529)
0924 00:12:22.600012 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.600047 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.600512 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.600765 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.600768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.601336 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:22.601547 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.601552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.602532 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.602533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.603484 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.603488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.604069 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.604070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.604213 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.604840 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:22.604848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:22.606394 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:22.608455 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.608463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.608466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.608824 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.609078 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.609079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.609079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.609236 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.609581 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:22.609589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:22.609777 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.609911 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.610128 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260924 00:12:22.611150 15942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:22.611282 15942 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:22.611482 15942 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:22.612134 15751 ts_manager.cc:194] Registered new tserver with Master: 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507)
I20260924 00:12:22.613109 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54529
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:12:22.630395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:22.712615 15947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:22.712839 15947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:22.712865 15947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:22.714026 15947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:22.714072 15947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:22.714099 15947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:12:22.715682 15947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:22.715963 15947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:22.716198 15947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:22.721290 15952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.721298 15953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.721526 15947 server_base.cc:1034] running on GCE node
W20260924 00:12:22.721302 15955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.721818 15947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:22.722076 15947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:22.723276 15947 hybrid_clock.cc:648] HybridClock initialized: now 1790208742723273 us; error 36 us; skew 500 ppm
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:22.725786 15947 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:12:22.726248 15947 webserver.cc:460] Webserver started at http://127.0.98.2:42679/ using document root <none> and password file <none>
I20260924 00:12:22.726490 15947 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:22.726558 15947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:22.726709 15947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:22.727821 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "0444e45deabf8675c69873d5a5119319"
server_key_iv: "11f280252a11fa20ddfd28b21f54c1be"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.728194 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "0444e45deabf8675c69873d5a5119319"
server_key_iv: "11f280252a11fa20ddfd28b21f54c1be"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.729485 15947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:12:22.730232 15962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.730394 15947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:22.730494 15947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "0444e45deabf8675c69873d5a5119319"
server_key_iv: "11f280252a11fa20ddfd28b21f54c1be"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.730574 15947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:22.752092 15947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:22.754220 15947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:22.754395 15947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:22.754639 15947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:22.755015 15947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:22.755074 15947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.755121 15947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:22.755155 15947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.761243 15947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36221
I20260924 00:12:22.761292 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36221 every 8 connection(s)
I20260924 00:12:22.761623 15947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:12:22.765872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 15947
I20260924 00:12:22.765997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:12:22.766222 392 external_mini_cluster.cc:1428] Setting key 2e6ece77c095ac5fecb259ff8f3bb933
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:22.772231 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.762324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:33421)
0924 00:12:22.762484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.762523 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.763082 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.763353 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.763356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.763940 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:22.764144 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.764149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.764861 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.764862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.765688 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.765690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.766113 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.766115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.766278 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.766939 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:22.766973 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:22.768820 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:22.770742 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.770749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.770751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.771020 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.771278 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.771279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.771279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.771391 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.771691 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:22.771695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:22.771815 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.771933 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.772063 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260924 00:12:22.772228 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.762338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33421 (local address 127.0.98.62:42949)
0924 00:12:22.762405 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.762408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.762607 (+ 199us) server_negotiation.cc:407] Connection header received
0924 00:12:22.763195 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.763197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.763209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.763275 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:22.764256 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.764759 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.765783 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.766030 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.766097 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.768948 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:22.768950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:22.768950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:22.768962 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:22.770654 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.771097 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.771098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.771099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.771165 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.771463 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.771464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.771465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.771622 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:22.771698 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.771996 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.772088 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:12:22.772833 16076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:22.772954 16076 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:22.773155 16076 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:22.773617 15751 ts_manager.cc:194] Registered new tserver with Master: 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
I20260924 00:12:22.774175 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33421
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:12:22.794466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:22.877043 16081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:22.877307 16081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:22.877336 16081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:22.878451 16081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:22.878489 16081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:22.878515 16081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:12:22.880038 16081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:22.880223 16081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:22.880398 16081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:22.885569 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.885596 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:22.885576 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:22.885947 16081 server_base.cc:1034] running on GCE node
I20260924 00:12:22.886119 16081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:22.886339 16081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:22.887442 16081 hybrid_clock.cc:648] HybridClock initialized: now 1790208742887412 us; error 44 us; skew 500 ppm
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:22.889866 16081 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:12:22.890350 16081 webserver.cc:460] Webserver started at http://127.0.98.3:45821/ using document root <none> and password file <none>
I20260924 00:12:22.890585 16081 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:22.890652 16081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:22.890785 16081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:12:22.891942 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "215b71672daa4a6ba644f39acc8f40b7"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "a4c1cf5e13159d1841c8b691abcd2fc7"
server_key_iv: "1995ea7f8a8c289ef7127a5637ca60ab"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.892324 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "215b71672daa4a6ba644f39acc8f40b7"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "a4c1cf5e13159d1841c8b691abcd2fc7"
server_key_iv: "1995ea7f8a8c289ef7127a5637ca60ab"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.894088 16081 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 00:12:22.894915 16096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.895114 16081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:22.895216 16081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "215b71672daa4a6ba644f39acc8f40b7"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "a4c1cf5e13159d1841c8b691abcd2fc7"
server_key_iv: "1995ea7f8a8c289ef7127a5637ca60ab"
server_key_version: "encryptionkey@0"
I20260924 00:12:22.895331 16081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:22.910447 16081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:22.913025 16081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:22.913198 16081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:22.913434 16081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:22.913753 16081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:22.913808 16081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.913851 16081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:22.913877 16081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:22.919996 16081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40945
I20260924 00:12:22.920043 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40945 every 8 connection(s)
I20260924 00:12:22.920377 16081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:12:22 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:22.929545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16081
I20260924 00:12:22.929646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:12:22.929850 392 external_mini_cluster.cc:1428] Setting key 8eebe574393fb7326be29cbb81e705ed
I20260924 00:12:22.930029 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.921176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48387 (local address 127.0.98.62:42949)
0924 00:12:22.921238 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.921240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.921421 (+ 181us) server_negotiation.cc:407] Connection header received
0924 00:12:22.921930 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.921931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.921941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.922005 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:22.923005 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.923797 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.924783 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.924981 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.925040 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.927058 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:22.927059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:22.927060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:22.927065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:22.928575 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.928923 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.928924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.928924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.928977 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.929225 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.929226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.929226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.929371 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:22.929434 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.929811 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.929880 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:12:22.930203 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.921149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:48387)
0924 00:12:22.921294 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.921335 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.921844 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.922062 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.922065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.922647 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:22.922895 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.922901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.923886 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.923886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.924670 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.924672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.925058 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.925058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.925155 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.925606 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:22.925613 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:22.926975 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:22.928643 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.928648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.928650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.928856 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.929035 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:22.929036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:22.929036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:22.929129 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:22.929469 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:22.929475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:22.929649 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.929772 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.929965 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 00:12:22.930718 16210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:22.930879 16210 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:22.931033 16210 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:22.931376 15751 ts_manager.cc:194] Registered new tserver with Master: 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945)
I20260924 00:12:22.931703 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48387
I20260924 00:12:22.941694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:12:22.948930 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.943016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.0.98.62:42949)
0924 00:12:22.943090 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.943092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.943190 (+ 98us) server_negotiation.cc:407] Connection header received
0924 00:12:22.943266 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.943267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.943278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.943331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:22.943864 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.944341 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.945054 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.945236 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.945286 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.946415 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:22.946416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:22.946416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:22.946422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:22.947858 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.948178 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.948179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.948180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.948228 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:22.948467 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:22.948468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:22.948468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:22.948607 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:22.948685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.948807 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.948865 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:12:22.950593 15751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:12:22.952214 15751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:12:22.958979 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36221 (local address 127.0.0.1:52308)
0924 00:12:22.954728 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.954773 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.954838 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.955158 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.955161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.955180 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:22.955284 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.955287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.956400 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.956401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.957777 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.957780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.958681 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.958682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.958738 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.958749 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.958837 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.958896 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260924 00:12:22.958987 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.98.2:36221)
0924 00:12:22.954982 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.954987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.954996 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:22.955040 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.955042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.955058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.955154 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:22.955360 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.956303 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.957868 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.958598 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.958675 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.958745 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.958862 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.958900 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260924 00:12:22.959928 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.0.1:39624)
0924 00:12:22.955117 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.955127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.955191 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.955489 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.955490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.955493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:22.955540 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.955542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.956642 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.956643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.957777 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.957780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.959682 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.959683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.959748 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.959752 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.959810 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.959857 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":7,"thread_start_us":90,"threads_started":1}
I20260924 00:12:22.959837 16010 tablet_service.cc:1467] Processing CreateTablet for tablet e5cee1a0e36d4f478b14683565025cc1 (DEFAULT_TABLE table=test-table [id=7da5478c22504072868f31f1e8a54122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:22.960167 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.0.98.3:40945)
0924 00:12:22.955073 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.955077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.955201 (+ 124us) server_negotiation.cc:407] Connection header received
0924 00:12:22.955301 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.955303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.955320 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.955397 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:22.955670 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.956527 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.958787 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.959574 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.959669 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.959745 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.960043 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.960087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260924 00:12:22.960235 16010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cee1a0e36d4f478b14683565025cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:22.960803 16144 tablet_service.cc:1467] Processing CreateTablet for tablet e5cee1a0e36d4f478b14683565025cc1 (DEFAULT_TABLE table=test-table [id=7da5478c22504072868f31f1e8a54122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:22.961119 16144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cee1a0e36d4f478b14683565025cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:22.963107 16227 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Bootstrap starting.
I20260924 00:12:22.963835 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44507 (local address 127.0.0.1:53736)
0924 00:12:22.955287 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:22.955297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:22.955348 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:22.956546 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:22.956548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:22.956550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:22.956602 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.956604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.957651 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.957652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.958725 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:22.958728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.963585 (+ 4857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.963586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:22.963653 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.963658 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:12:22.963715 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:12:22.963760 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":76,"threads_started":1}
I20260924 00:12:22.963848 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:22.954879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.0.98.1:44507)
0924 00:12:22.955046 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:22.955051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:22.956398 (+ 1347us) server_negotiation.cc:407] Connection header received
0924 00:12:22.956439 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:22.956441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:22.956453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:22.956535 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:22.956657 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.957569 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.958812 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:22.963476 (+ 4664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:22.963581 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:22.963639 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:22.963730 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:12:22.963768 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260924 00:12:22.964246 16227 tablet_bootstrap.cc:654] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:22.964329 16228 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Bootstrap starting.
I20260924 00:12:22.964421 15876 tablet_service.cc:1467] Processing CreateTablet for tablet e5cee1a0e36d4f478b14683565025cc1 (DEFAULT_TABLE table=test-table [id=7da5478c22504072868f31f1e8a54122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:22.964666 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cee1a0e36d4f478b14683565025cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:22.965034 16228 tablet_bootstrap.cc:654] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:22.965044 16227 log.cc:824] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:22.965560 16228 log.cc:824] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:22.965946 16227 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: No bootstrap required, opened a new log
I20260924 00:12:22.966024 16227 ts_tablet_manager.cc:1397] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 00:12:22.966507 16228 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: No bootstrap required, opened a new log
I20260924 00:12:22.966589 16228 ts_tablet_manager.cc:1397] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:22.967717 16231 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap starting.
I20260924 00:12:22.968147 16227 raft_consensus.cc:348] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.968317 16227 raft_consensus.cc:374] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:22.968359 16227 raft_consensus.cc:729] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7951d4ba0ea949f3b48bc7ade99900a6, State: Initialized, Role: FOLLOWER
I20260924 00:12:22.968402 16231 tablet_bootstrap.cc:654] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:22.968287 16228 raft_consensus.cc:348] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.968456 16228 raft_consensus.cc:374] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:22.968508 16228 raft_consensus.cc:729] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215b71672daa4a6ba644f39acc8f40b7, State: Initialized, Role: FOLLOWER
I20260924 00:12:22.968493 16227 consensus_queue.cc:260] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.968768 16227 ts_tablet_manager.cc:1428] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 00:12:22.968799 16076 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
I20260924 00:12:22.968946 16231 log.cc:824] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:22.968688 16228 consensus_queue.cc:260] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.969255 16228 ts_tablet_manager.cc:1428] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:12:22.969367 16210 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
I20260924 00:12:22.969946 16231 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: No bootstrap required, opened a new log
I20260924 00:12:22.970018 16231 ts_tablet_manager.cc:1397] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:22.971652 16231 raft_consensus.cc:348] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.971782 16231 raft_consensus.cc:374] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:22.971805 16231 raft_consensus.cc:729] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f85ff0f31c4a51af1a3b9aaa024c1b, State: Initialized, Role: FOLLOWER
I20260924 00:12:22.971938 16231 consensus_queue.cc:260] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:22.972195 16231 ts_tablet_manager.cc:1428] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:12:22.972236 15942 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
W20260924 00:12:23.012209 16077 tablet.cc:2369] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:12:23.058924 16232 raft_consensus.cc:482] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:23.059051 16232 raft_consensus.cc:504] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:23.059481 16232 leader_election.cc:288] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507), 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945)
I20260924 00:12:23.063537 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.059729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44773 (local address 127.0.98.1:44507)
0924 00:12:23.059780 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:23.059782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:23.059826 (+ 44us) server_negotiation.cc:407] Connection header received
0924 00:12:23.059858 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:23.059859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:23.059871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:23.059922 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:23.060146 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.060955 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.062542 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.063147 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.063265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:23.063278 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:23.063423 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:12:23.063464 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:12:23.063537 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.059769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36903 (local address 127.0.98.3:40945)
0924 00:12:23.059812 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:23.059814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:23.059993 (+ 179us) server_negotiation.cc:407] Connection header received
0924 00:12:23.060030 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:23.060031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:23.060040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:23.060083 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:23.060244 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.060940 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.062543 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.063147 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.063274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:23.063285 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:23.063423 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 00:12:23.063464 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:12:23.063576 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.059653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44507 (local address 127.0.98.2:44773)
0924 00:12:23.059728 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:23.059740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:23.059786 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:23.059959 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:23.059960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:23.059962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:23.060042 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:23.060044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.061039 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.061040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:23.062388 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:23.062391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.063248 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.063249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:23.063334 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:23.063341 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:12:23.063400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:12:23.063449 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":67}
I20260924 00:12:23.063587 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.059699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.98.2:36903)
0924 00:12:23.059880 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:23.059889 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:23.059969 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:23.060108 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:23.060109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:23.060111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:23.060169 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:23.060170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.061039 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.061040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:23.062388 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:23.062391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.063249 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.063250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:23.063334 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:23.063341 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:12:23.063400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:12:23.063449 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":84,"thread_start_us":121,"threads_started":1}
I20260924 00:12:23.064002 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cee1a0e36d4f478b14683565025cc1" candidate_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215b71672daa4a6ba644f39acc8f40b7" is_pre_election: true
I20260924 00:12:23.063979 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cee1a0e36d4f478b14683565025cc1" candidate_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" is_pre_election: true
I20260924 00:12:23.064183 16164 raft_consensus.cc:2454] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7951d4ba0ea949f3b48bc7ade99900a6 in term 0.
I20260924 00:12:23.064183 15896 raft_consensus.cc:2454] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7951d4ba0ea949f3b48bc7ade99900a6 in term 0.
I20260924 00:12:23.064424 15964 leader_election.cc:302] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6; no voters:
I20260924 00:12:23.064581 16232 raft_consensus.cc:2781] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:12:23.064638 16232 raft_consensus.cc:482] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:12:23.064661 16232 raft_consensus.cc:3037] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:23.065394 16232 raft_consensus.cc:504] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:23.065547 16232 leader_election.cc:288] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [CANDIDATE]: Term 1 election: Requested vote from peers 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507), 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945)
I20260924 00:12:23.065717 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cee1a0e36d4f478b14683565025cc1" candidate_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215b71672daa4a6ba644f39acc8f40b7"
I20260924 00:12:23.065753 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cee1a0e36d4f478b14683565025cc1" candidate_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
I20260924 00:12:23.065810 16164 raft_consensus.cc:3037] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:23.065821 15896 raft_consensus.cc:3037] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:23.066717 16164 raft_consensus.cc:2454] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7951d4ba0ea949f3b48bc7ade99900a6 in term 1.
I20260924 00:12:23.066717 15896 raft_consensus.cc:2454] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7951d4ba0ea949f3b48bc7ade99900a6 in term 1.
I20260924 00:12:23.066938 15966 leader_election.cc:302] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b, 7951d4ba0ea949f3b48bc7ade99900a6; no voters:
I20260924 00:12:23.067075 16232 raft_consensus.cc:2781] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:23.067294 16232 raft_consensus.cc:686] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7951d4ba0ea949f3b48bc7ade99900a6, State: Running, Role: LEADER
I20260924 00:12:23.067446 16232 consensus_queue.cc:237] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:23.068356 15751 catalog_manager.cc:5495] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } health_report { overall_health: UNKNOWN } } }
I20260924 00:12:23.073513 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.071096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.2:36221)
0924 00:12:23.071146 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:23.071148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:23.071158 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:12:23.071244 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:23.071245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:23.071264 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:23.071346 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:23.071492 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.071960 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.072723 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:23.072930 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:23.073014 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:23.073186 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:23.073284 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:12:23.073376 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:23.073415 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:12:23.073434 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:12:23.075186 16164 raft_consensus.cc:1261] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Refusing update from remote peer 7951d4ba0ea949f3b48bc7ade99900a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:23.075191 15896 raft_consensus.cc:1261] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Refusing update from remote peer 7951d4ba0ea949f3b48bc7ade99900a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:23.075548 16232 consensus_queue.cc:1035] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:23.075716 16237 consensus_queue.cc:1035] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:23.078121 16241 mvcc.cc:205] Tried to move back new op lower bound from 7332695011633684480 to 7332695011605139456. Current Snapshot: MvccSnapshot[applied={T|T < 7332695011633684480}]
I20260924 00:12:23.078269 16240 mvcc.cc:205] Tried to move back new op lower bound from 7332695011633684480 to 7332695011605139456. Current Snapshot: MvccSnapshot[applied={T|T < 7332695011633684480 or (T in {7332695011633684480})}]
I20260924 00:12:23.080235 15750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:23.080339 15750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:23.081157 15750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Sending DeleteTablet for 3 replicas of tablet e5cee1a0e36d4f478b14683565025cc1
I20260924 00:12:23.081517 16010 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cee1a0e36d4f478b14683565025cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52308
I20260924 00:12:23.081585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15813
I20260924 00:12:23.081575 15876 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cee1a0e36d4f478b14683565025cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53736
I20260924 00:12:23.084134 16144 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cee1a0e36d4f478b14683565025cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39624
I20260924 00:12:23.084604 16251 tablet_replica.cc:330] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: stopping tablet replica
I20260924 00:12:23.084728 16251 raft_consensus.cc:2229] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:23.084843 16251 raft_consensus.cc:2258] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:23.085323 16251 ts_tablet_manager.cc:1905] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:12:23.087355 15738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44507 (error 108)
I20260924 00:12:23.087601 16250 tablet_replica.cc:330] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: stopping tablet replica
I20260924 00:12:23.087698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15947
I20260924 00:12:23.087733 16250 raft_consensus.cc:2229] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:12:23.087863 16250 raft_consensus.cc:2258] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:23.088747 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.087867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44507 (local address 127.0.0.1:53744)
0924 00:12:23.088673 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:23.088717 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784}
W20260924 00:12:23.088910 15738 catalog_manager.cc:4584] TS 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5cee1a0e36d4f478b14683565025cc1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44507: connect: Connection refused (error 111)
I20260924 00:12:23.092033 16251 ts_tablet_manager.cc:1918] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:23.092130 16251 log.cc:1196] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/e5cee1a0e36d4f478b14683565025cc1
I20260924 00:12:23.092419 16251 ts_tablet_manager.cc:1939] T e5cee1a0e36d4f478b14683565025cc1 P 215b71672daa4a6ba644f39acc8f40b7: Deleting consensus metadata
I20260924 00:12:23.092922 15736 catalog_manager.cc:4841] TS 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): tablet e5cee1a0e36d4f478b14683565025cc1 (table test-table [id=7da5478c22504072868f31f1e8a54122]) successfully deleted
I20260924 00:12:23.093756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16081
I20260924 00:12:23.093807 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:23.093689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36221 (local address 127.0.0.1:52328)
0924 00:12:23.093745 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:23.093777 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 00:12:23.098572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 15719
I20260924 00:12:43.104270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--webserver_interface=127.0.98.62
--webserver_port=36023
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:12:43.189781 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:43.189996 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:43.190019 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:43.191120 16253 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:12:43.191157 16253 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:12:43.191171 16253 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:12:43.191183 16253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:12:43.191196 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:43.191242 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:43.191294 16253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:12:43.191325 16253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:12:43.192776 16253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:43.192967 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:43.193135 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:12:43.201990 16253 server_base.cc:1034] running on GCE node
W20260924 00:12:43.202028 16260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.202119 16262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.202028 16259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:43.202644 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:43.203020 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:43.204195 16253 hybrid_clock.cc:648] HybridClock initialized: now 1790208763204178 us; error 44 us; skew 500 ppm
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:43.207106 16253 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:12:43.207624 16253 webserver.cc:460] Webserver started at http://127.0.98.62:36023/ using document root <none> and password file <none>
I20260924 00:12:43.207875 16253 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:43.207929 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:43.209336 16253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:43.210026 16269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.210201 16253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:12:43.210297 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7dfefa502e81463484c132f33fb7608e"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "b81dd85a8329b6b2e62a18386f852137"
server_key_iv: "100d919084a5d0dc2a97bde7498cc2d0"
server_key_version: "encryptionkey@0"
I20260924 00:12:43.210685 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:43.246431 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:43.248847 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:43.248961 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:43.252900 16253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42949
I20260924 00:12:43.252955 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42949 every 8 connection(s)
I20260924 00:12:43.253356 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:12:43.254876 16322 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:12:43.255622 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap starting.
I20260924 00:12:43.260035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16253
I20260924 00:12:43.260358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44507
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=45581
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:43.264554 16322 log.cc:824] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:43.266466 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:43.266762 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap complete.
I20260924 00:12:43.269110 16322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:43.269349 16322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.269524 16322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:43.269586 16322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:12:43.269649 16322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:12:43.269724 16322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:12:43.270773 16322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:43.270860 16322 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfefa502e81463484c132f33fb7608e; no voters:
I20260924 00:12:43.271078 16322 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:12:43.271114 16326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:12:43.271306 16326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 LEADER]: Becoming Leader. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Running, Role: LEADER
I20260924 00:12:43.271384 16322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:12:43.271450 16326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:12:43.271741 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfefa502e81463484c132f33fb7608e. Latest consensus state: current_term: 2 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:12:43.271746 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:12:43.271847 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:43.271876 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:12:43.272171 16333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:12:43.273012 16333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:12:43.273115 16333 catalog_manager.cc:1259] Loaded cluster ID: 7427c371d4004dfbb08209071f575b3c
I20260924 00:12:43.273146 16333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:12:43.274326 16333 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:12:43.274510 16333 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Loaded TSK: 0
I20260924 00:12:43.288846 16333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Generated new TSK 1
I20260924 00:12:43.288952 16333 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:12:43.344812 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:43.345052 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:43.345089 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:43.346252 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:43.346321 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:43.346367 16324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:12:43.347886 16324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45581
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:43.348155 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:43.348392 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:43.353334 16349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.353346 16350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.353356 16352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:43.353564 16324 server_base.cc:1034] running on GCE node
I20260924 00:12:43.353823 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:43.354063 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:43.355269 16324 hybrid_clock.cc:648] HybridClock initialized: now 1790208763355243 us; error 42 us; skew 500 ppm
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:43.357805 16324 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:12:43.358299 16324 webserver.cc:460] Webserver started at http://127.0.98.1:45581/ using document root <none> and password file <none>
I20260924 00:12:43.358533 16324 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:43.358588 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:43.360082 16324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.360805 16359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.360975 16324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:43.361076 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "e5339ebab4416d95cbbcc92c26bea56c"
server_key_iv: "d1a37261f606f7f98c27df5f8a543ab0"
server_key_version: "encryptionkey@0"
I20260924 00:12:43.361425 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:43.375041 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:43.377565 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:43.377799 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:43.378113 16324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:43.378634 16366 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:12:43.380188 16324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:12:43.380251 16324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:12:43.380285 16324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:12:43.380956 16324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:12:43.380999 16324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.381043 16366 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap starting.
I20260924 00:12:43.388674 16324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44507
I20260924 00:12:43.389220 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:12:43.389820 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44507 every 8 connection(s)
I20260924 00:12:43.392562 16366 log.cc:824] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:43.394928 16366 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:43.395330 16366 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap complete.
I20260924 00:12:43.396013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16324
I20260924 00:12:43.396298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36221
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=42679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:43.396559 16366 ts_tablet_manager.cc:1397] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260924 00:12:43.398000 16366 raft_consensus.cc:348] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:43.398378 16366 raft_consensus.cc:729] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f85ff0f31c4a51af1a3b9aaa024c1b, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.398553 16366 consensus_queue.cc:260] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:43.398860 16366 ts_tablet_manager.cc:1428] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:43.406013 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.389973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:59931)
0924 00:12:43.390200 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.390254 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.391034 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.393268 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.393272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.393640 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:43.393917 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.393924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.395645 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.395646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.397340 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.397343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.398710 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.398711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.398842 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.399521 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:43.399532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:43.401419 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:43.404012 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.404019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.404021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.404336 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.404649 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.404650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.404650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.404779 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.405347 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:43.405354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:43.405511 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.405617 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.405795 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20260924 00:12:43.406490 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.390027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59931 (local address 127.0.98.62:42949)
0924 00:12:43.392806 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.392816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.392862 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:12:43.392999 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.393006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.393061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.393206 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.394077 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.395519 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.397458 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.398315 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.398588 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.401567 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.401573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.401573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.401587 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.403914 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.404432 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.404436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.404436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.404563 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.404905 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.404906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.404907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.405257 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.405345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.405704 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.405862 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2713,"thread_start_us":81,"threads_started":1}
I20260924 00:12:43.407282 16474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:43.407387 16474 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:43.407583 16474 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:43.408391 16286 ts_manager.cc:194] Registered new tserver with Master: 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507)
I20260924 00:12:43.409461 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59931
I20260924 00:12:43.410449 16474 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
I20260924 00:12:43.414970 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.409241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.0.98.1:44507)
0924 00:12:43.411558 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.411562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.411574 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:12:43.411617 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.411619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.411631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.411693 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:43.411840 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.412536 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.414068 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.414610 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.414669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.414705 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.414892 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.414921 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2267,"thread_start_us":77,"threads_started":1}
I20260924 00:12:43.415040 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.409003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44507 (local address 127.0.0.1:33358)
0924 00:12:43.411334 (+ 2331us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.411484 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.411550 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.411705 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.411708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.411710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:43.411768 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.411770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.412613 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.412614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.413970 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.413973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.414684 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.414685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.414778 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.414793 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.414875 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.414939 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2300,"thread_start_us":76,"threads_started":1}
I20260924 00:12:43.415467 16408 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cee1a0e36d4f478b14683565025cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33358
I20260924 00:12:43.416055 16483 tablet_replica.cc:330] stopping tablet replica
I20260924 00:12:43.416178 16483 raft_consensus.cc:2229] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:43.416260 16483 raft_consensus.cc:2258] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:43.416597 16483 ts_tablet_manager.cc:1905] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:43.418746 16483 ts_tablet_manager.cc:1918] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:43.418834 16483 log.cc:1196] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal/wals/e5cee1a0e36d4f478b14683565025cc1
I20260924 00:12:43.419083 16483 ts_tablet_manager.cc:1939] T e5cee1a0e36d4f478b14683565025cc1 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Deleting consensus metadata
I20260924 00:12:43.419597 16273 catalog_manager.cc:4841] TS 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507): tablet e5cee1a0e36d4f478b14683565025cc1 (table test-table [id=7da5478c22504072868f31f1e8a54122]) successfully deleted
W20260924 00:12:43.484488 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:43.484706 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:43.484733 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:43.485839 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:43.485877 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:43.485905 16479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:12:43.487407 16479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42679
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:43.487609 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:43.487845 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:43.493147 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.493160 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.493176 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:43.493362 16479 server_base.cc:1034] running on GCE node
I20260924 00:12:43.493580 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:43.493855 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:43.495034 16479 hybrid_clock.cc:648] HybridClock initialized: now 1790208763495010 us; error 38 us; skew 500 ppm
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:43.497814 16479 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:12:43.498301 16479 webserver.cc:460] Webserver started at http://127.0.98.2:42679/ using document root <none> and password file <none>
I20260924 00:12:43.498551 16479 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:43.498607 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:43.500061 16479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.500756 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.501084 16479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:12:43.501169 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "0444e45deabf8675c69873d5a5119319"
server_key_iv: "11f280252a11fa20ddfd28b21f54c1be"
server_key_version: "encryptionkey@0"
I20260924 00:12:43.501518 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:43.511543 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:43.514040 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:43.514201 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:43.514444 16479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:43.514887 16505 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:12:43.516443 16479 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:12:43.516515 16479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:12:43.516566 16479 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:12:43.517213 16479 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:12:43.517269 16479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.517352 16505 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Bootstrap starting.
I20260924 00:12:43.524675 16479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36221
I20260924 00:12:43.524791 16612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36221 every 8 connection(s)
I20260924 00:12:43.525116 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:12:43.527038 16505 log.cc:824] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:43.529158 16505 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:12:43.529467 16505 tablet_bootstrap.cc:492] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Bootstrap complete.
I20260924 00:12:43.530392 16505 ts_tablet_manager.cc:1397] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:43.531472 16505 raft_consensus.cc:348] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:43.531731 16505 raft_consensus.cc:729] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7951d4ba0ea949f3b48bc7ade99900a6, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.531903 16505 consensus_queue.cc:260] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } } peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 00:12:43.532195 16505 ts_tablet_manager.cc:1428] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20260924 00:12:43.532481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16479
I20260924 00:12:43.532738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40945
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=45821
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:12:43.535385 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.525980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59113 (local address 127.0.98.62:42949)
0924 00:12:43.526053 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.526056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.526413 (+ 357us) server_negotiation.cc:407] Connection header received
0924 00:12:43.527124 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.527126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.527144 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.527262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.527912 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.528426 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.529518 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.529792 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.529868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.532257 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.532259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.532259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.532267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.533959 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.534356 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.534357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.534358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.534423 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.534688 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.534689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.534689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.534814 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.534884 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.535164 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.535292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:12:43.535561 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.525882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:59113)
0924 00:12:43.526222 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.526286 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.526988 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.527335 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.527339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.527614 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:43.527804 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.527809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.528524 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.528525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.529426 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.529429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.529888 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.529889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.530021 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.530600 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:43.530608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:43.532157 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:43.534027 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.534034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.534036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.534274 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.534501 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.534502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.534502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.534614 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.534886 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:43.534894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:43.535035 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.535145 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.535351 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":102,"threads_started":1}
I20260924 00:12:43.536096 16613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:43.536203 16613 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:43.536414 16613 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:43.537027 16286 ts_manager.cc:194] Registered new tserver with Master: 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
I20260924 00:12:43.537483 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59113
I20260924 00:12:43.539379 16613 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 00:12:43.545264 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.537346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36221 (local address 127.0.0.1:44466)
0924 00:12:43.537459 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.537470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.537513 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.537967 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.537968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.538403 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:43.538453 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.538456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.539385 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.539386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.539934 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.539936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.540411 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.540414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.540474 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.541000 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:43.541005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:43.542298 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:43.544120 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.544128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.544130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.544332 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.544557 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.544558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.544558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.544623 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.544913 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:43.544920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:43.544972 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.545057 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.545163 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 00:12:43.545295 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.537593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.98.2:36221)
0924 00:12:43.537782 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.537787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.537798 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:12:43.537854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.537856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.537870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.537945 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.538543 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.539300 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.540019 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.540331 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.540413 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.542389 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.542397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.542397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.542404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.544015 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.544418 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.544421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.544421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.544481 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.544685 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.544686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.544687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.544834 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.544920 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.545105 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.545206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260924 00:12:43.545646 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cee1a0e36d4f478b14683565025cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44466
I20260924 00:12:43.546142 16620 tablet_replica.cc:330] stopping tablet replica
I20260924 00:12:43.546267 16620 raft_consensus.cc:2229] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:43.546336 16620 raft_consensus.cc:2258] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:43.546638 16620 ts_tablet_manager.cc:1905] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:43.548874 16620 ts_tablet_manager.cc:1918] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:43.548938 16620 log.cc:1196] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/e5cee1a0e36d4f478b14683565025cc1
I20260924 00:12:43.549154 16620 ts_tablet_manager.cc:1939] T e5cee1a0e36d4f478b14683565025cc1 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting consensus metadata
I20260924 00:12:43.549631 16273 catalog_manager.cc:4841] TS 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): tablet e5cee1a0e36d4f478b14683565025cc1 (table test-table [id=7da5478c22504072868f31f1e8a54122]) successfully deleted
W20260924 00:12:43.620636 16618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:12:43.620891 16618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:12:43.620934 16618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:12:43.622205 16618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:12:43.622282 16618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:12:43.622334 16618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:12:43.624014 16618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45821
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:12:43.624305 16618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:12:43.624565 16618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:12:43.630091 16625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:43.630159 16618 server_base.cc:1034] running on GCE node
W20260924 00:12:43.630095 16626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:12:43.630108 16628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:12:43.630550 16618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:12:43.630815 16618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:12:43.631982 16618 hybrid_clock.cc:648] HybridClock initialized: now 1790208763631961 us; error 47 us; skew 500 ppm
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:12:43.634563 16618 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:12:43.634933 16618 webserver.cc:460] Webserver started at http://127.0.98.3:45821/ using document root <none> and password file <none>
I20260924 00:12:43.635167 16618 fs_manager.cc:359] Metadata directory not provided
I20260924 00:12:43.635277 16618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:12:43.636708 16618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:43.637347 16635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.637545 16618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:43.637617 16618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "215b71672daa4a6ba644f39acc8f40b7"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "a4c1cf5e13159d1841c8b691abcd2fc7"
server_key_iv: "1995ea7f8a8c289ef7127a5637ca60ab"
server_key_version: "encryptionkey@0"
I20260924 00:12:43.638003 16618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:12:43.648221 16618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:12:43.650467 16618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:12:43.650625 16618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:12:43.650906 16618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:12:43.651314 16618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:12:43.651353 16618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.651374 16618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:12:43.651402 16618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:12:43.657387 16618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40945
I20260924 00:12:43.657428 16748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40945 every 8 connection(s)
I20260924 00:12:43.657757 16618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:12:43.658185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16618
Sep 24 00:12:43 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:12:43.666076 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.658563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.0.98.62:42949)
0924 00:12:43.658652 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.658653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.658930 (+ 277us) server_negotiation.cc:407] Connection header received
0924 00:12:43.659005 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.659006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.659019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.659091 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.659774 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.660576 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.661402 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.661683 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.661749 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.663067 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.663069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.663069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.663076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.664739 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.665142 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.665143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.665144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.665216 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.665477 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.665478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.665479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.665608 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.665727 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.665878 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.665964 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":59,"threads_started":1}
I20260924 00:12:43.668386 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.658543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47147 (local address 127.0.98.62:42949)
0924 00:12:43.658597 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.658599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.658882 (+ 283us) server_negotiation.cc:407] Connection header received
0924 00:12:43.659536 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.659537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.659548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.659604 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.660784 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.661238 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.662557 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.662757 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.662828 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.665529 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.665531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.665531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.665538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.667281 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.667639 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.667640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.667640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.667691 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.667882 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.667883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.667883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.667979 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.668035 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.668243 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.668325 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:12:43.668529 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.658544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:47147)
0924 00:12:43.658754 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.658804 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.659441 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.659681 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.659685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.660416 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:12:43.660686 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.660692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.661328 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.661328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.662456 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.662458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.662848 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.662849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.662975 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.663506 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:12:43.663517 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:12:43.665409 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:12:43.667363 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.667369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.667371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.667575 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.667741 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:12:43.667741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:12:43.667742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:12:43.667828 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:12:43.668028 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:12:43.668031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:12:43.668133 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.668209 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.668323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260924 00:12:43.669040 16749 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42949
I20260924 00:12:43.669139 16749 heartbeater.cc:464] Registering TS with master...
I20260924 00:12:43.669291 16749 heartbeater.cc:511] Master 127.0.98.62:42949 requested a full tablet report, sending...
I20260924 00:12:43.669656 16286 ts_manager.cc:194] Registered new tserver with Master: 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945)
I20260924 00:12:43.670056 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47147
I20260924 00:12:43.677801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:12:43.680531 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.678176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.0.98.62:42949)
0924 00:12:43.678219 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.678221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.678311 (+ 90us) server_negotiation.cc:407] Connection header received
0924 00:12:43.678377 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.678378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.678408 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.678466 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:43.678640 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.679116 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.679966 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.680137 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.680209 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.680338 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:43.680376 (+ 38us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 00:12:43.680487 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50240: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 00:12:43.680603 16477 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50240: token expired
W20260924 00:12:43.680733 16214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42949)
I20260924 00:12:43.686492 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.681002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.98.62:42949)
0924 00:12:43.681034 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.681035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.681046 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:12:43.681102 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.681103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.681109 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.681149 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:12:43.681624 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.682064 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.682744 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.682941 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.682990 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.684189 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:12:43.684190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:12:43.684191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:12:43.684196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:12:43.685578 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.685908 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.685908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.685909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.685959 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:12:43.686125 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:12:43.686126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:12:43.686126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:12:43.686214 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:12:43.686286 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.686386 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.686437 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 00:12:43.687748 16286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:12:43.689579 16286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:12:43.691834 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 1916956691bf4f91993db3724ac05d65 (DEFAULT_TABLE table=test-table [id=fe9df78a09ad443d91b3f400943377dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:43.691835 16408 tablet_service.cc:1467] Processing CreateTablet for tablet 1916956691bf4f91993db3724ac05d65 (DEFAULT_TABLE table=test-table [id=fe9df78a09ad443d91b3f400943377dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:43.692039 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916956691bf4f91993db3724ac05d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:43.692039 16408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916956691bf4f91993db3724ac05d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:43.693744 16366 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap starting.
I20260924 00:12:43.694088 16366 tablet_bootstrap.cc:654] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:43.694358 16505 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Bootstrap starting.
I20260924 00:12:43.694727 16505 tablet_bootstrap.cc:654] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:43.694836 16366 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: No bootstrap required, opened a new log
I20260924 00:12:43.694898 16366 ts_tablet_manager.cc:1397] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.695026 16366 raft_consensus.cc:348] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.695133 16366 raft_consensus.cc:374] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:43.695197 16366 raft_consensus.cc:729] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f85ff0f31c4a51af1a3b9aaa024c1b, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.695269 16366 consensus_queue.cc:260] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.695367 16505 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: No bootstrap required, opened a new log
I20260924 00:12:43.695384 16366 ts_tablet_manager.cc:1428] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:12:43.695416 16505 ts_tablet_manager.cc:1397] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 00:12:43.695794 16505 raft_consensus.cc:348] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.695878 16505 raft_consensus.cc:374] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:43.695935 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.691566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.0.1:56672)
0924 00:12:43.691667 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.691678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.691717 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.692058 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.692059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.692061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:43.692114 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.692115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.693302 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.693303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.694303 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.694306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.695667 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.695668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.695742 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.695746 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.695813 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.695865 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 00:12:43.695927 16505 raft_consensus.cc:729] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7951d4ba0ea949f3b48bc7ade99900a6, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.696022 16505 consensus_queue.cc:260] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.696127 16505 ts_tablet_manager.cc:1428] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:12:43.696281 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.691679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.98.3:40945)
0924 00:12:43.691864 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.691869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.691878 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:12:43.691937 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.691940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.691956 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.692181 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:43.692267 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.693202 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.694496 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.695552 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.695658 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.695741 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.696131 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.696184 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260924 00:12:43.696996 16683 tablet_service.cc:1467] Processing CreateTablet for tablet 1916956691bf4f91993db3724ac05d65 (DEFAULT_TABLE table=test-table [id=fe9df78a09ad443d91b3f400943377dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:12:43.697278 16683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916956691bf4f91993db3724ac05d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:12:43.699985 16758 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Bootstrap starting.
I20260924 00:12:43.700573 16758 tablet_bootstrap.cc:654] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:12:43.701018 16758 log.cc:824] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:12:43.701850 16758 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: No bootstrap required, opened a new log
I20260924 00:12:43.701923 16758 ts_tablet_manager.cc:1397] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:12:43.702596 16480 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:12:43.702719 16480 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.703047 16480 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
I20260924 00:12:43.703724 16758 raft_consensus.cc:348] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.703910 16758 raft_consensus.cc:374] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:12:43.704592 16758 raft_consensus.cc:729] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215b71672daa4a6ba644f39acc8f40b7, State: Initialized, Role: FOLLOWER
I20260924 00:12:43.704725 16758 consensus_queue.cc:260] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.705013 16758 ts_tablet_manager.cc:1428] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:12:43.705075 16749 heartbeater.cc:503] Master 127.0.98.62:42949 was elected leader, sending a full tablet report...
I20260924 00:12:43.706614 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.703249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.98.1:39879)
0924 00:12:43.703306 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.703317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.703359 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.703588 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.703589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.703591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:43.703639 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.703640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.704531 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.704532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.705676 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.705678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.706370 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.706371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.706435 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.706438 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.706486 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.706531 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:12:43.706815 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.703398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39879 (local address 127.0.98.3:40945)
0924 00:12:43.703440 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.703442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.703452 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:12:43.703473 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.703474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.703483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.703541 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:43.703835 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.704442 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.705780 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.706307 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.706373 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.706382 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.706726 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.706765 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:12:43.707072 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.703249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36221 (local address 127.0.98.1:46511)
0924 00:12:43.703562 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.703571 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:12:43.703647 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:12:43.703788 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:12:43.703789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:12:43.703790 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:12:43.704192 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.704195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.704918 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.704919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.705890 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:12:43.705893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.706818 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.706819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:12:43.706904 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.706906 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:12:43.706950 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 00:12:43.706987 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":159,"thread_start_us":108,"threads_started":1}
I20260924 00:12:43.707039 16703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916956691bf4f91993db3724ac05d65" candidate_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215b71672daa4a6ba644f39acc8f40b7" is_pre_election: true
I20260924 00:12:43.707089 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.703370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46511 (local address 127.0.98.2:36221)
0924 00:12:43.703412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.703415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.703636 (+ 221us) server_negotiation.cc:407] Connection header received
0924 00:12:43.703700 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.703701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.703722 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.703787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:12:43.704272 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.704850 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.705990 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.706631 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.706673 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.706760 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.706977 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.707022 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:12:43.707175 16703 raft_consensus.cc:2454] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f85ff0f31c4a51af1a3b9aaa024c1b in term 0.
I20260924 00:12:43.707352 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916956691bf4f91993db3724ac05d65" candidate_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" is_pre_election: true
I20260924 00:12:43.707394 16361 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215b71672daa4a6ba644f39acc8f40b7, 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters:
I20260924 00:12:43.707507 16567 raft_consensus.cc:2454] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f85ff0f31c4a51af1a3b9aaa024c1b in term 0.
I20260924 00:12:43.707505 16480 raft_consensus.cc:2781] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:12:43.707571 16480 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:12:43.707607 16480 raft_consensus.cc:3037] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:43.708254 16480 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.708403 16480 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 1 election: Requested vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
I20260924 00:12:43.708545 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916956691bf4f91993db3724ac05d65" candidate_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
I20260924 00:12:43.708623 16567 raft_consensus.cc:3037] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:43.709280 16567 raft_consensus.cc:2454] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f85ff0f31c4a51af1a3b9aaa024c1b in term 1.
I20260924 00:12:43.709371 16703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916956691bf4f91993db3724ac05d65" candidate_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215b71672daa4a6ba644f39acc8f40b7"
I20260924 00:12:43.709456 16703 raft_consensus.cc:3037] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:12:43.709486 16363 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b, 7951d4ba0ea949f3b48bc7ade99900a6; no voters:
I20260924 00:12:43.709600 16480 raft_consensus.cc:2781] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:12:43.709785 16480 raft_consensus.cc:686] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 LEADER]: Becoming Leader. State: Replica: 43f85ff0f31c4a51af1a3b9aaa024c1b, State: Running, Role: LEADER
I20260924 00:12:43.709858 16480 consensus_queue.cc:237] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:12:43.710421 16703 raft_consensus.cc:2454] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f85ff0f31c4a51af1a3b9aaa024c1b in term 1.
I20260924 00:12:43.710682 16286 catalog_manager.cc:5495] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } health_report { overall_health: HEALTHY } } }
I20260924 00:12:43.719882 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.717809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.0.98.1:44507)
0924 00:12:43.717844 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:12:43.717846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:12:43.717852 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:12:43.717881 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:12:43.717882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:12:43.717897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:12:43.717958 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:12:43.718057 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.718511 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.719205 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:12:43.719450 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:12:43.719504 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:12:43.719626 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:12:43.719715 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:12:43.719762 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:12:43.719812 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 00:12:43.719828 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:12:43.721334 16567 raft_consensus.cc:1261] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Refusing update from remote peer 43f85ff0f31c4a51af1a3b9aaa024c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:43.721362 16703 raft_consensus.cc:1261] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Refusing update from remote peer 43f85ff0f31c4a51af1a3b9aaa024c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:12:43.721710 16480 consensus_queue.cc:1035] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:43.721853 16762 consensus_queue.cc:1035] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:12:43.724061 16765 mvcc.cc:205] Tried to move back new op lower bound from 7332695096200290304 to 7332695096156180480. Current Snapshot: MvccSnapshot[applied={T|T < 7332695096200290304}]
I20260924 00:12:43.724288 16767 mvcc.cc:205] Tried to move back new op lower bound from 7332695096200290304 to 7332695096156180480. Current Snapshot: MvccSnapshot[applied={T|T < 7332695096200290304}]
I20260924 00:12:43.724593 16766 mvcc.cc:205] Tried to move back new op lower bound from 7332695096200290304 to 7332695096156180480. Current Snapshot: MvccSnapshot[applied={T|T < 7332695096200290304}]
I20260924 00:12:43.726444 16285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:43.726544 16285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:12:43.727334 16285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Sending DeleteTablet for 3 replicas of tablet 1916956691bf4f91993db3724ac05d65
I20260924 00:12:43.727690 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916956691bf4f91993db3724ac05d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44466
I20260924 00:12:43.727734 16683 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916956691bf4f91993db3724ac05d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:12:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56672
I20260924 00:12:43.727725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16324
I20260924 00:12:43.727794 16620 tablet_replica.cc:330] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: stopping tablet replica
I20260924 00:12:43.727846 16620 raft_consensus.cc:2229] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:43.727901 16620 raft_consensus.cc:2258] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:43.728564 16775 tablet_replica.cc:330] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: stopping tablet replica
I20260924 00:12:43.728636 16775 raft_consensus.cc:2229] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:12:43.728708 16775 raft_consensus.cc:2258] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:12:43.728698 16620 ts_tablet_manager.cc:1905] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:43.729070 16775 ts_tablet_manager.cc:1905] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:12:43.730530 16620 ts_tablet_manager.cc:1918] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:43.730557 16775 ts_tablet_manager.cc:1918] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:12:43.730582 16620 log.cc:1196] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/1916956691bf4f91993db3724ac05d65
I20260924 00:12:43.730608 16775 log.cc:1196] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/1916956691bf4f91993db3724ac05d65
I20260924 00:12:43.730803 16620 ts_tablet_manager.cc:1939] T 1916956691bf4f91993db3724ac05d65 P 7951d4ba0ea949f3b48bc7ade99900a6: Deleting consensus metadata
I20260924 00:12:43.730844 16775 ts_tablet_manager.cc:1939] T 1916956691bf4f91993db3724ac05d65 P 215b71672daa4a6ba644f39acc8f40b7: Deleting consensus metadata
I20260924 00:12:43.731251 16271 catalog_manager.cc:4841] TS 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): tablet 1916956691bf4f91993db3724ac05d65 (table test-table [id=fe9df78a09ad443d91b3f400943377dd]) successfully deleted
I20260924 00:12:43.731495 16273 catalog_manager.cc:4841] TS 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): tablet 1916956691bf4f91993db3724ac05d65 (table test-table [id=fe9df78a09ad443d91b3f400943377dd]) successfully deleted
W20260924 00:12:43.734108 16273 connection.cc:535] client connection to 127.0.98.1:44507 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 00:12:43.734206 16273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:12:43.734542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16479
I20260924 00:12:43.734704 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:12:43.734563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44507 (local address 127.0.0.1:33374)
0924 00:12:43.734626 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:12:43.734667 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:12:43.735773 16273 catalog_manager.cc:4584] TS 43f85ff0f31c4a51af1a3b9aaa024c1b (127.0.98.1:44507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1916956691bf4f91993db3724ac05d65: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44507: connect: Connection refused (error 111)
I20260924 00:12:43.739269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16618
I20260924 00:12:43.744103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16253
I20260924 00:13:03.750617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--webserver_interface=127.0.98.62
--webserver_port=36023
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:03.830678 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:03.830936 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:03.830973 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:03.832203 16777 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:13:03.832273 16777 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:13:03.832296 16777 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:13:03.832314 16777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:03.832350 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:03.832381 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:03.832409 16777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:03.832435 16777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:03.833926 16777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:03.834203 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:03.834429 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:03.839555 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:03.839562 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:03.839560 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:03.839685 16777 server_base.cc:1034] running on GCE node
I20260924 00:13:03.840101 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:03.840405 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:03.841588 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790208783841576 us; error 34 us; skew 500 ppm
Sep 24 00:13:03 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:03.844561 16777 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:03.845067 16777 webserver.cc:460] Webserver started at http://127.0.98.62:36023/ using document root <none> and password file <none>
I20260924 00:13:03.845304 16777 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:03.845396 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:03.846760 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:03.847537 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:03.847718 16777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:03.847826 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7dfefa502e81463484c132f33fb7608e"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "b81dd85a8329b6b2e62a18386f852137"
server_key_iv: "100d919084a5d0dc2a97bde7498cc2d0"
server_key_version: "encryptionkey@0"
I20260924 00:13:03.848187 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:03.876088 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:03.878535 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:03.878705 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:03.883139 16777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42949
I20260924 00:13:03.883191 16844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42949 every 8 connection(s)
I20260924 00:13:03.883769 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:03.885264 16845 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:13:03.885957 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap starting.
I20260924 00:13:03.885963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16777
I20260924 00:13:03.886307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44507
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=45581
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:03.894503 16845 log.cc:824] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:03.897060 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:13:03.897418 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Bootstrap complete.
I20260924 00:13:03.899757 16845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:13:03.900007 16845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Initialized, Role: FOLLOWER
I20260924 00:13:03.900221 16845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:13:03.900317 16845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:03.900372 16845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:03.900419 16845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:13:03.901391 16845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:13:03.901515 16845 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfefa502e81463484c132f33fb7608e; no voters:
I20260924 00:13:03.901737 16845 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:13:03.901772 16849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:13:03.901882 16849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [term 3 LEADER]: Becoming Leader. State: Replica: 7dfefa502e81463484c132f33fb7608e, State: Running, Role: LEADER
I20260924 00:13:03.901966 16849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260924 00:13:03.902004 16845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:03.902216 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfefa502e81463484c132f33fb7608e. Latest consensus state: current_term: 3 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:13:03.902280 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:03.902491 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7dfefa502e81463484c132f33fb7608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfefa502e81463484c132f33fb7608e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260924 00:13:03.902772 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:03.902493 16853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:03.903735 16853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:03.903903 16853 catalog_manager.cc:1259] Loaded cluster ID: 7427c371d4004dfbb08209071f575b3c
I20260924 00:13:03.904215 16853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:03.905627 16853 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:03.905879 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Loaded TSK: 0
I20260924 00:13:03.905975 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7dfefa502e81463484c132f33fb7608e: Loaded TSK: 1
F20260924 00:13:03.906308 16853 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc2d198a9ad google::LogMessage::Fail() at ??:0
@ 0x7fc2d198d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc2d198a4de google::LogMessage::Flush() at ??:0
@ 0x7fc2d198ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc2d40f7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc2d40f4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc2d98e81cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc2d98e848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc2d98e8594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc2d98aec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc2d98b8ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc2d38580d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc2d384d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc2d46b16db start_thread at ??:0
@ 0x7fc2d048861f clone at ??:0
W20260924 00:13:03.993180 16847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:03.993422 16847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:03.993461 16847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:03.994688 16847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:03.994757 16847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:03.994804 16847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:03.996325 16847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45581
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:03.996593 16847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:03.996830 16847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:04.001874 16886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:04.001955 16847 server_base.cc:1034] running on GCE node
W20260924 00:13:04.001888 16887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:04.002068 16889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:04.002306 16847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:04.002569 16847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:04.003755 16847 hybrid_clock.cc:648] HybridClock initialized: now 1790208784003731 us; error 38 us; skew 500 ppm
Sep 24 00:13:04 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:04.006772 16847 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:13:04.007367 16847 webserver.cc:460] Webserver started at http://127.0.98.1:45581/ using document root <none> and password file <none>
I20260924 00:13:04.007685 16847 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:04.007761 16847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:04.009569 16847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:04.010386 16896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.010599 16847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:04.010694 16847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "e5339ebab4416d95cbbcc92c26bea56c"
server_key_iv: "d1a37261f606f7f98c27df5f8a543ab0"
server_key_version: "encryptionkey@0"
I20260924 00:13:04.011165 16847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:04.033804 16847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:04.036628 16847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:04.036810 16847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:04.037115 16847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:04.037761 16903 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:13:04.039393 16847 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:13:04.039456 16847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:13:04.039512 16847 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:13:04.040565 16847 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:13:04.040616 16847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:13:04.040716 16903 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap starting.
I20260924 00:13:04.048763 16847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44507
I20260924 00:13:04.048990 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44507 every 8 connection(s)
I20260924 00:13:04.049281 16847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:13:04.051965 16903 log.cc:824] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:04.054104 16903 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:13:04.054419 16903 tablet_bootstrap.cc:492] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Bootstrap complete.
I20260924 00:13:04.054886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 16847
I20260924 00:13:04.055182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36221
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=42679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:04.055691 16903 ts_tablet_manager.cc:1397] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260924 00:13:04.057174 16903 raft_consensus.cc:348] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:04.057433 16903 raft_consensus.cc:729] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f85ff0f31c4a51af1a3b9aaa024c1b, State: Initialized, Role: FOLLOWER
I20260924 00:13:04.057581 16903 consensus_queue.cc:260] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:04.057906 16903 ts_tablet_manager.cc:1428] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 00:13:04.130282 17013 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:04.050003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:39067)
0924 00:13:04.050189 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.050235 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:04.050974 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:04.130044 (+ 79070us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
W20260924 00:13:04.130585 17011 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42949: Client connection negotiation failed: client connection to 127.0.98.62:42949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:13:04.131069 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.130944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:45841)
0924 00:13:04.131007 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.131048 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 00:13:04.131433 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.131341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:57597)
0924 00:13:04.131392 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.131416 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 00:13:04.131574 17011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:13:04.138928 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:04.139122 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:04.139145 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:04.140326 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:04.140372 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:04.140398 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:04.141844 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42679
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:04.142076 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:04.142282 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:04.147500 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:04.147519 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:04.147533 17021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:04.147744 17015 server_base.cc:1034] running on GCE node
I20260924 00:13:04.148029 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:04.148231 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:04.149372 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790208784149356 us; error 27 us; skew 500 ppm
Sep 24 00:13:04 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:04.151991 17015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:13:04.152472 17015 webserver.cc:460] Webserver started at http://127.0.98.2:42679/ using document root <none> and password file <none>
I20260924 00:13:04.152755 17015 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:04.152806 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:04.154151 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:04.154871 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.155077 17015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:04.155170 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "7951d4ba0ea949f3b48bc7ade99900a6"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "0444e45deabf8675c69873d5a5119319"
server_key_iv: "11f280252a11fa20ddfd28b21f54c1be"
server_key_version: "encryptionkey@0"
I20260924 00:13:04.155587 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:04.163041 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:04.165480 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:04.165648 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:04.165900 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:04.166285 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:04.166342 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.166388 17015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:04.166409 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.172804 17015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36221
I20260924 00:13:04.172847 17144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36221 every 8 connection(s)
I20260924 00:13:04.173182 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:04.174252 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.173831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:32873)
0924 00:13:04.174013 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.174112 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260924 00:13:04.174669 17145 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42949: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
I20260924 00:13:04.174978 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.174884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:34787)
0924 00:13:04.174931 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.174959 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:13:04.175335 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.175263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:40043)
0924 00:13:04.175290 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.175315 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 00:13:04.175449 17145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:13:04.180951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17015
I20260924 00:13:04.181210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40945
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42949
--webserver_port=45821
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:04.261338 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:04.261543 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:04.261565 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:04.262722 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:04.262765 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:04.262791 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:04.264252 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45821
--tserver_master_addrs=127.0.98.62:42949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:04.264494 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:04.264729 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:04.269786 17153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:04.269870 17148 server_base.cc:1034] running on GCE node
W20260924 00:13:04.269773 17154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:04.269763 17156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:04.270143 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:04.270393 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:04.271552 17148 hybrid_clock.cc:648] HybridClock initialized: now 1790208784271510 us; error 54 us; skew 500 ppm
Sep 24 00:13:04 dist-test-slave-3q8k krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:04.274151 17148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:13:04.274628 17148 webserver.cc:460] Webserver started at http://127.0.98.3:45821/ using document root <none> and password file <none>
I20260924 00:13:04.274917 17148 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:04.274993 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:04.276465 17148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:04.277206 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.277412 17148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:04.277505 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "215b71672daa4a6ba644f39acc8f40b7"
format_stamp: "Formatted at 2026-09-24 00:12:22 on dist-test-slave-3q8k"
server_key: "a4c1cf5e13159d1841c8b691abcd2fc7"
server_key_iv: "1995ea7f8a8c289ef7127a5637ca60ab"
server_key_version: "encryptionkey@0"
I20260924 00:13:04.277895 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:04.299849 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:04.302266 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:04.302436 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:04.302685 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:04.302990 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:04.303047 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.303086 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:04.303114 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:04.309357 17148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40945
I20260924 00:13:04.309405 17276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40945 every 8 connection(s)
I20260924 00:13:04.309768 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:13:04.310858 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.310445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:49179)
0924 00:13:04.310613 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.310717 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260924 00:13:04.311364 17277 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42949 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42949: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
I20260924 00:13:04.311720 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.311641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:56577)
0924 00:13:04.311677 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.311703 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 00:13:04.312057 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:04.312001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:60487)
0924 00:13:04.312025 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:04.312044 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 00:13:04.312199 17277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:13:04.316262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17148
I20260924 00:13:05.132206 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.131945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:41339)
0924 00:13:05.132088 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:05.132137 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260924 00:13:05.176002 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.175794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:37689)
0924 00:13:05.175912 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:05.175949 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260924 00:13:05.312728 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.312518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:38479)
0924 00:13:05.312641 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:05.312679 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260924 00:13:05.551856 17284 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:05.551977 17284 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:05.552273 17284 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
Sep 24 00:13:05 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 00:13:05 dist-test-slave-3q8k krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 00:13:05.563179 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.552709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41863 (local address 127.0.98.2:36221)
0924 00:13:05.552903 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:05.552911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:05.552939 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:13:05.553448 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:05.553450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:05.553494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:05.553624 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:05.554415 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.555331 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556321 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.556743 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556863 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:05.559446 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:05.559450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:05.559450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:05.559461 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:05.561388 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:05.561860 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:05.561863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:05.561863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:05.561945 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:05.562195 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:05.562196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:05.562196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:05.562533 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:05.562616 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:05.562937 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 00:13:05.563069 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260924 00:13:05.563174 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.552463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36221 (local address 127.0.98.1:41863)
0924 00:13:05.552723 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:05.552734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:05.552773 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:05.553766 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:05.553767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:05.554070 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:05.554279 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:05.554287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.555430 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.555431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:05.556220 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:05.556222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.556837 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:05.556947 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:05.557571 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:05.557580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:05.559254 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:05.561480 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:05.561486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:05.561487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:05.561779 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:05.562011 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:05.562012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:05.562012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:05.562129 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:05.562604 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:05.562616 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:05.562745 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 00:13:05.562862 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 00:13:05.563058 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":158,"thread_start_us":62,"threads_started":1}
I20260924 00:13:05.563199 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.552604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54163 (local address 127.0.98.3:40945)
0924 00:13:05.552742 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:05.552756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:05.552792 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:13:05.553400 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:05.553404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:05.553463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:05.553655 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:05.554417 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.555358 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556319 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.556744 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556863 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:05.559723 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:05.559726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:05.559726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:05.559736 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:05.561709 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:05.562064 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:05.562067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:05.562067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:05.562189 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:05.562401 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:05.562402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:05.562402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:05.562681 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:05.562749 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:05.562937 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 00:13:05.563069 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260924 00:13:05.563915 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:05.552463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.98.1:54163)
0924 00:13:05.552537 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:05.552551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:05.552626 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:05.553697 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:05.553700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:05.554086 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:05.554279 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:05.554285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.555456 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.555457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:05.556222 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:05.556224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:05.556835 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:05.556836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:05.556955 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:05.557596 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:05.557600 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:05.559622 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:05.561787 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:05.561788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:05.561788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:05.561964 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:05.562285 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:05.562286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:05.562286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:05.562334 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:05.562755 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:05.562756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:05.562790 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 00:13:05.562862 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:13:05.563135 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":160}
W20260924 00:13:05.564102 16900 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
W20260924 00:13:05.564594 16898 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
I20260924 00:13:05.564651 16898 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6
I20260924 00:13:05.564776 17284 raft_consensus.cc:2726] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:13:06.132990 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:06.132760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:40477)
0924 00:13:06.132902 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:06.132943 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260924 00:13:06.176697 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:06.176486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:53353)
0924 00:13:06.176614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:06.176656 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 00:13:06.313485 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:06.313275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:33531)
0924 00:13:06.313404 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:06.313444 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260924 00:13:07.133760 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:07.133507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:53107)
0924 00:13:07.133668 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:07.133711 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260924 00:13:07.177409 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:07.177176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:41123)
0924 00:13:07.177320 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:07.177366 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260924 00:13:07.239308 17293 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:07.239411 17293 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:07.239621 17293 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
W20260924 00:13:07.240058 16898 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
W20260924 00:13:07.240154 16900 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
I20260924 00:13:07.240255 16900 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6
I20260924 00:13:07.240345 17293 raft_consensus.cc:2726] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:13:07.314231 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:07.314017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:40383)
0924 00:13:07.314150 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:07.314190 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260924 00:13:08.134720 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:08.134503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:33859)
0924 00:13:08.134632 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:08.134674 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 00:13:08.178117 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:08.177910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:58747)
0924 00:13:08.178037 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:08.178075 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260924 00:13:08.314885 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:08.314669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:52913)
0924 00:13:08.314804 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:08.314845 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 00:13:08.887053 17298 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:08.887181 17298 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:08.887434 17298 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
W20260924 00:13:08.887800 16900 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
W20260924 00:13:08.887897 16898 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
I20260924 00:13:08.887996 16898 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6
I20260924 00:13:08.888087 17298 raft_consensus.cc:2726] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:13:09.135564 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:09.135295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:46599)
0924 00:13:09.135432 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:09.135470 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260924 00:13:09.178848 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:09.178586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:60005)
0924 00:13:09.178754 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:09.178799 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260924 00:13:09.315641 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:09.315430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:44579)
0924 00:13:09.315546 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:09.315599 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260924 00:13:10.136328 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:10.136069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:40555)
0924 00:13:10.136238 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:10.136281 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260924 00:13:10.179584 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:10.179356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:41381)
0924 00:13:10.179500 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:10.179539 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260924 00:13:10.316314 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:10.316088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:60919)
0924 00:13:10.316230 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:10.316270 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 00:13:10.471468 17305 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:10.471598 17305 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:10.471810 17305 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
W20260924 00:13:10.472239 16900 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
W20260924 00:13:10.472366 16898 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
I20260924 00:13:10.472443 16898 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6
I20260924 00:13:10.472561 17305 raft_consensus.cc:2726] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:13:11.137133 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:11.136899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:54425)
0924 00:13:11.137034 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:11.137086 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260924 00:13:11.180244 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:11.180035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:41795)
0924 00:13:11.180164 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:11.180200 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"threads_started":1}
I20260924 00:13:11.317155 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:11.316937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:35105)
0924 00:13:11.317065 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:11.317112 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 00:13:12.137851 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:12.137606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:43787)
0924 00:13:12.137754 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:12.137797 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260924 00:13:12.180860 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:12.180642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:50771)
0924 00:13:12.180768 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:12.180814 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 00:13:12.317870 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:12.317663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:44603)
0924 00:13:12.317787 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:12.317826 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 00:13:13.138622 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:13.138372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:42429)
0924 00:13:13.138532 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:13.138574 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260924 00:13:13.181556 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:13.181320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:51095)
0924 00:13:13.181468 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:13.181508 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260924 00:13:13.318624 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:13.318408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:38175)
0924 00:13:13.318545 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:13.318580 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260924 00:13:14.139487 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:14.139198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:53417)
0924 00:13:14.139397 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:14.139440 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":113,"threads_started":1}
I20260924 00:13:14.182232 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:14.182009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:32771)
0924 00:13:14.182148 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:14.182189 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260924 00:13:14.319738 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:14.319255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:51821)
0924 00:13:14.319657 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:14.319693 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":334,"threads_started":1}
I20260924 00:13:15.140329 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:15.140066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:38749)
0924 00:13:15.140228 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:15.140272 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 00:13:15.182967 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:15.182694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:42017)
0924 00:13:15.182840 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:15.182906 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260924 00:13:15.320526 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:15.320286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:37763)
0924 00:13:15.320447 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:15.320481 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260924 00:13:16.141103 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:16.140868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:36943)
0924 00:13:16.141013 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:16.141055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260924 00:13:16.183751 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:16.183535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:35081)
0924 00:13:16.183660 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:16.183704 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260924 00:13:16.321249 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:16.321010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:52085)
0924 00:13:16.321161 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:16.321202 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 00:13:17.141927 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:17.141682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:48227)
0924 00:13:17.141832 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:17.141877 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260924 00:13:17.184619 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:17.184343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:46427)
0924 00:13:17.184503 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:17.184552 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":114,"threads_started":1}
I20260924 00:13:17.322036 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:17.321776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:35745)
0924 00:13:17.321939 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:17.321984 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260924 00:13:17.857380 17327 raft_consensus.cc:482] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:17.857493 17327 raft_consensus.cc:504] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215b71672daa4a6ba644f39acc8f40b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } } peers { permanent_uuid: "7951d4ba0ea949f3b48bc7ade99900a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36221 } } peers { permanent_uuid: "43f85ff0f31c4a51af1a3b9aaa024c1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44507 } }
I20260924 00:13:17.857720 17327 leader_election.cc:288] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945), 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221)
W20260924 00:13:17.858193 16900 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7951d4ba0ea949f3b48bc7ade99900a6 (127.0.98.2:36221): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
W20260924 00:13:17.858316 16898 leader_election.cc:341] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 215b71672daa4a6ba644f39acc8f40b7 (127.0.98.3:40945): Not found: Tablet not found: 1916956691bf4f91993db3724ac05d65
I20260924 00:13:17.858392 16898 leader_election.cc:302] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f85ff0f31c4a51af1a3b9aaa024c1b; no voters: 215b71672daa4a6ba644f39acc8f40b7, 7951d4ba0ea949f3b48bc7ade99900a6
I20260924 00:13:17.858460 17327 raft_consensus.cc:2726] T 1916956691bf4f91993db3724ac05d65 P 43f85ff0f31c4a51af1a3b9aaa024c1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:13:18.142701 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:18.142466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:54859)
0924 00:13:18.142604 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:18.142651 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260924 00:13:18.185410 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:18.185173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:33627)
0924 00:13:18.185318 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:18.185365 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 00:13:18.322873 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:18.322558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:42877)
0924 00:13:18.322723 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:18.322795 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260924 00:13:19.143585 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:19.143349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.1:53987)
0924 00:13:19.143495 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:19.143541 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260924 00:13:19.186234 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:19.185997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.2:50661)
0924 00:13:19.186144 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:19.186186 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 00:13:19.323800 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:19.323549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42949 (local address 127.0.98.3:59725)
0924 00:13:19.323708 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:19.323751 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260924 00:13:19.325955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 16847
I20260924 00:13:19.330941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17015
I20260924 00:13:19.335662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17148
2026-09-24T00:13:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58222 ms)
[----------] 30 tests from SecurityITest (231148 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:19 dist-test-slave-3q8k krb5kdc[17336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:19 dist-test-slave-3q8k krb5kdc[17336](info): set up 2 sockets
Sep 24 00:13:19 dist-test-slave-3q8k krb5kdc[17336](info): commencing operation
krb5kdc: starting...
W20260924 00:13:20.390861 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:20.518050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:20.601542 17349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:20.601800 17349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:20.601840 17349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:20.603011 17349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:20.603097 17349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:20.603123 17349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:20.603147 17349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:20.603215 17349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:20.604894 17349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:20.605109 17349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:20.605289 17349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:20.610656 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:20.610677 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:20.610901 17349 server_base.cc:1034] running on GCE node
W20260924 00:13:20.610667 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:20.611356 17349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:20.611624 17349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:20.612785 17349 hybrid_clock.cc:648] HybridClock initialized: now 1790208800612769 us; error 31 us; skew 500 ppm
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:20.615398 17349 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:20.615906 17349 webserver.cc:460] Webserver started at http://127.0.98.62:43431/ using document root <none> and password file <none>
I20260924 00:13:20.616156 17349 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:20.616227 17349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:20.616372 17349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:20.617445 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "3503948536b74bde862c49d3940be6f3"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "082192b3951596e5fa3dbef9116801ba"
server_key_iv: "b42c045328ddbf704e10ddde5b218622"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.617820 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "3503948536b74bde862c49d3940be6f3"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "082192b3951596e5fa3dbef9116801ba"
server_key_iv: "b42c045328ddbf704e10ddde5b218622"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.619138 17349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:20.619933 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:20.620112 17349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:20.620225 17349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "3503948536b74bde862c49d3940be6f3"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "082192b3951596e5fa3dbef9116801ba"
server_key_iv: "b42c045328ddbf704e10ddde5b218622"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.620301 17349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:20.644733 17349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:20.646934 17349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:20.647122 17349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:20.651432 17349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41643
I20260924 00:13:20.651497 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41643 every 8 connection(s)
I20260924 00:13:20.651888 17349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:20.652854 17417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:20.653347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17349
I20260924 00:13:20.653434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:20.653657 392 external_mini_cluster.cc:1428] Setting key 220bb899bf3fbccfd01794d33b422b90
I20260924 00:13:20.656172 17417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: Bootstrap starting.
I20260924 00:13:20.656808 17417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:20.657115 17417 log.cc:824] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:20.657881 17417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: No bootstrap required, opened a new log
I20260924 00:13:20.660516 17417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260924 00:13:20.660665 17417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:20.660687 17417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3503948536b74bde862c49d3940be6f3, State: Initialized, Role: FOLLOWER
I20260924 00:13:20.660825 17417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260924 00:13:20.660905 17417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:20.660928 17417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:20.661002 17417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:20.661625 17417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260924 00:13:20.661721 17417 leader_election.cc:302] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3503948536b74bde862c49d3940be6f3; no voters:
I20260924 00:13:20.661933 17417 leader_election.cc:288] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:20.661964 17422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:20.662074 17422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3503948536b74bde862c49d3940be6f3, State: Running, Role: LEADER
I20260924 00:13:20.662166 17422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } }
I20260924 00:13:20.662209 17417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:20.662492 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3503948536b74bde862c49d3940be6f3. Latest consensus state: current_term: 1 leader_uuid: "3503948536b74bde862c49d3940be6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } } }
I20260924 00:13:20.662587 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:20.662757 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3503948536b74bde862c49d3940be6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3503948536b74bde862c49d3940be6f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41643 } } }
I20260924 00:13:20.662825 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:20.663120 17431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:20.663646 17431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:20.665715 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:20.654337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.0.98.62:41643)
0924 00:13:20.654519 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:20.654527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:20.654553 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:13:20.654958 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:20.654964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:20.655012 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:20.655143 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:20.655978 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.657023 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.658055 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.658519 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.658603 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:20.661596 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:20.661600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:20.661600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:20.661612 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:20.664043 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:20.664537 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:20.664542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:20.664542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:20.664658 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:20.664882 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:20.664883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:20.664884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:20.665140 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:20.665310 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:20.665409 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 00:13:20.665547 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260924 00:13:20.668411 17431 catalog_manager.cc:1347] Generated new cluster ID: 33690f4b0ddf49799deff7d994a97c2b
I20260924 00:13:20.668491 17431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:20.691195 17431 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:20.691782 17431 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:20.695124 17431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: Generated new TSK 0
I20260924 00:13:20.695333 17431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:13:20.726666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:20.821918 17443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:20.822155 17443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:20.822196 17443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:20.823527 17443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:20.823576 17443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:20.823604 17443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:20.825116 17443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:20.825325 17443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:20.825516 17443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:20.830765 17451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:20.830754 17449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:20.830757 17448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:20.831192 17443 server_base.cc:1034] running on GCE node
I20260924 00:13:20.831462 17443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:20.831727 17443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:20.832895 17443 hybrid_clock.cc:648] HybridClock initialized: now 1790208800832872 us; error 40 us; skew 500 ppm
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:20.835592 17443 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:13:20.836162 17443 webserver.cc:460] Webserver started at http://127.0.98.1:43395/ using document root <none> and password file <none>
I20260924 00:13:20.836410 17443 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:20.836485 17443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:20.836639 17443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:20.837734 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "0093518a501244b6b78876a92d7cd702"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "fa6084c1aae559d83919664ffd0d9d11"
server_key_iv: "51deff0b86383112c95aafd0932f7316"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.838106 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0093518a501244b6b78876a92d7cd702"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "fa6084c1aae559d83919664ffd0d9d11"
server_key_iv: "51deff0b86383112c95aafd0932f7316"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.839568 17443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 00:13:20.840457 17458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:20.840672 17443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:20.840782 17443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "0093518a501244b6b78876a92d7cd702"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "fa6084c1aae559d83919664ffd0d9d11"
server_key_iv: "51deff0b86383112c95aafd0932f7316"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.840859 17443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:20.856208 17443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:20.858382 17443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:20.858551 17443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:20.858788 17443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:20.859105 17443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:20.859162 17443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:20.859234 17443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:20.859266 17443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:20.865819 17443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37133
I20260924 00:13:20.865911 17571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37133 every 8 connection(s)
I20260924 00:13:20.866264 17443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:13:20.872681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17443
I20260924 00:13:20.872804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:20.873026 392 external_mini_cluster.cc:1428] Setting key d04aaeeb80cf73f213334c65d727b73b
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:20.876843 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:20.867111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57933 (local address 127.0.98.62:41643)
0924 00:13:20.867256 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:20.867259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:20.867455 (+ 196us) server_negotiation.cc:407] Connection header received
0924 00:13:20.868456 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:20.868458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:20.868471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:20.868529 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:20.869211 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.869770 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.870853 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.871070 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.871158 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:20.873733 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:20.873735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:20.873735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:20.873744 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:20.875422 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:20.875818 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:20.875820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:20.875820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:20.875886 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:20.876128 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:20.876129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:20.876129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:20.876274 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:20.876344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:20.876589 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 00:13:20.876697 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 00:13:20.876863 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:20.867042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41643 (local address 127.0.98.1:57933)
0924 00:13:20.867316 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:20.867365 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:20.868336 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:20.868610 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:20.868621 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:20.868903 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:20.869106 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:20.869110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.869882 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.869883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:20.870747 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:20.870749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:20.871175 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:20.871176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:20.871356 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:20.871866 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:20.871874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:20.873609 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:20.875503 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:20.875509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:20.875511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:20.875742 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:20.875958 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:20.875959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:20.875959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:20.876062 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:20.876339 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:20.876344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:20.876460 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 00:13:20.876549 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:13:20.876696 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
I20260924 00:13:20.877434 17572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41643
I20260924 00:13:20.877544 17572 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:20.877754 17572 heartbeater.cc:511] Master 127.0.98.62:41643 requested a full tablet report, sending...
I20260924 00:13:20.878324 17381 ts_manager.cc:194] Registered new tserver with Master: 0093518a501244b6b78876a92d7cd702 (127.0.98.1:37133)
I20260924 00:13:20.878959 17381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57933
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:13:20.900470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:20.983896 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:20.984105 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:20.984129 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:20.985226 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:20.985263 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:20.985289 17577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:20.986706 17577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:20.986892 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:20.987061 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:20.991982 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:20.992013 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:20.992039 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:20.992229 17577 server_base.cc:1034] running on GCE node
I20260924 00:13:20.992859 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:20.993064 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:20.994237 17577 hybrid_clock.cc:648] HybridClock initialized: now 1790208800994218 us; error 31 us; skew 500 ppm
Sep 24 00:13:20 dist-test-slave-3q8k krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:20.996819 17577 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:13:20.997327 17577 webserver.cc:460] Webserver started at http://127.0.98.2:44429/ using document root <none> and password file <none>
I20260924 00:13:20.997555 17577 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:20.997623 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:20.997776 17577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:20.998795 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc845975e4c9464195a61024cd7e9c26"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "3232a5e0fda97cad6c4bf1a11793e2b3"
server_key_iv: "f094af6ed05f6e1d89d748004ba1bea2"
server_key_version: "encryptionkey@0"
I20260924 00:13:20.999150 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc845975e4c9464195a61024cd7e9c26"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "3232a5e0fda97cad6c4bf1a11793e2b3"
server_key_iv: "f094af6ed05f6e1d89d748004ba1bea2"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.000474 17577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:21.001235 17592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.001430 17577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:21.001526 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "dc845975e4c9464195a61024cd7e9c26"
format_stamp: "Formatted at 2026-09-24 00:13:20 on dist-test-slave-3q8k"
server_key: "3232a5e0fda97cad6c4bf1a11793e2b3"
server_key_iv: "f094af6ed05f6e1d89d748004ba1bea2"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.001610 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.016386 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.018922 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.019099 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:21.019397 17577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:21.019726 17577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:21.019781 17577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.019825 17577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:21.019855 17577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.026099 17577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40087
I20260924 00:13:21.026185 17705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40087 every 8 connection(s)
I20260924 00:13:21.026507 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:13:21 dist-test-slave-3q8k krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:21.036625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17577
I20260924 00:13:21.036738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:21.036957 392 external_mini_cluster.cc:1428] Setting key 18188fcad78356874661db8b3db9c899
I20260924 00:13:21.037107 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.027787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46191 (local address 127.0.98.62:41643)
0924 00:13:21.027863 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.027866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.027878 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:13:21.028742 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.028744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.028758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.028851 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.029503 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.030376 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.031498 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.031870 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.031941 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.034207 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.034208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.034209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:21.034215 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.035709 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.036099 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.036100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.036100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.036153 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.036370 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.036371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.036371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.036515 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.036567 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.036924 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.037019 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:13:21.037432 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.027337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41643 (local address 127.0.98.2:46191)
0924 00:13:21.027519 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.027563 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.028631 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.028905 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.028908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.029168 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:21.029418 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.029423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.030479 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.030480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.031416 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.031419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.031961 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.031962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.032113 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.032660 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:21.032668 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:21.034073 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:21.035786 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:21.035792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:21.035794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:21.036019 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:21.036214 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:21.036215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:21.036215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:21.036316 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:21.036592 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:21.036599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:21.036779 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.036909 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.037165 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260924 00:13:21.038012 17706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41643
I20260924 00:13:21.038112 17706 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:21.038306 17706 heartbeater.cc:511] Master 127.0.98.62:41643 requested a full tablet report, sending...
I20260924 00:13:21.038800 17381 ts_manager.cc:194] Registered new tserver with Master: dc845975e4c9464195a61024cd7e9c26 (127.0.98.2:40087)
I20260924 00:13:21.039168 17381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:13:21.062772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:21.143847 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:21.144095 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:21.144131 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:21.145403 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:21.145483 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:21.145529 17711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:21.147075 17711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:21.147392 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:21.147641 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:21.153057 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.153131 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.153155 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.153357 17711 server_base.cc:1034] running on GCE node
I20260924 00:13:21.153581 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:21.153824 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:21.154970 17711 hybrid_clock.cc:648] HybridClock initialized: now 1790208801154980 us; error 59 us; skew 500 ppm
Sep 24 00:13:21 dist-test-slave-3q8k krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:21.157660 17711 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:13:21.158215 17711 webserver.cc:460] Webserver started at http://127.0.98.3:37455/ using document root <none> and password file <none>
I20260924 00:13:21.158452 17711 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:21.158499 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:21.158661 17711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:21.159849 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "96816f783c6b4bc28c997c835a55075e"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "6e7a5d86141154a514de52906a33e8a8"
server_key_iv: "4a8ca63cbfc740a4ccf3c6879c9f2e0b"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.160228 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "96816f783c6b4bc28c997c835a55075e"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "6e7a5d86141154a514de52906a33e8a8"
server_key_iv: "4a8ca63cbfc740a4ccf3c6879c9f2e0b"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.161665 17711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:21.162585 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.162808 17711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:21.162890 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "96816f783c6b4bc28c997c835a55075e"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "6e7a5d86141154a514de52906a33e8a8"
server_key_iv: "4a8ca63cbfc740a4ccf3c6879c9f2e0b"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.162985 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.179996 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.182372 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.182542 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:21.182794 17711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:21.183121 17711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:21.183176 17711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.183245 17711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:21.183276 17711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.189353 17711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37615
I20260924 00:13:21.189400 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37615 every 8 connection(s)
I20260924 00:13:21.189775 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:13:21 dist-test-slave-3q8k krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:21.197845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17711
I20260924 00:13:21.197969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:21.198148 392 external_mini_cluster.cc:1428] Setting key 445077ac3e3b7e8f3ef478ba4019c282
I20260924 00:13:21.200198 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.190645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42005 (local address 127.0.98.62:41643)
0924 00:13:21.190732 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.190734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.190944 (+ 210us) server_negotiation.cc:407] Connection header received
0924 00:13:21.191981 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.191982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.191995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.192065 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.192719 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.193220 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.194264 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.194461 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.194550 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.196954 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.196955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.196955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:21.196963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.198871 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.199394 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.199395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.199395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.199444 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.199669 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.199670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.199670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.199762 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.199806 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.200029 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.200136 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:13:21.200349 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.190626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41643 (local address 127.0.98.3:42005)
0924 00:13:21.190818 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.190854 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.191868 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.192149 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.192153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.192422 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:21.192611 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.192615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.193329 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.193330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.194149 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.194151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.194559 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.194560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.194687 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.195358 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:21.195366 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:21.196864 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:21.198972 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:21.198979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:21.198981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:21.199332 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:21.199503 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:21.199504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:21.199504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:21.199626 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:21.199811 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:21.199815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:21.199926 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.200000 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.200124 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260924 00:13:21.200799 17840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41643
I20260924 00:13:21.200894 17840 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:21.201030 17840 heartbeater.cc:511] Master 127.0.98.62:41643 requested a full tablet report, sending...
I20260924 00:13:21.201357 17381 ts_manager.cc:194] Registered new tserver with Master: 96816f783c6b4bc28c997c835a55075e (127.0.98.3:37615)
I20260924 00:13:21.201746 17381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42005
I20260924 00:13:21.210080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:21.218325 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.211455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.0.98.62:41643)
0924 00:13:21.211501 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.211504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.211622 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:13:21.211704 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.211705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.211715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.211787 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.212583 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.213079 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.213853 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.214063 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.214342 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.215679 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.215681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.215681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:21.215688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.217239 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.217561 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.217561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.217562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.217613 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:21.217826 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:21.217827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:21.217827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:21.217964 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.218039 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.218187 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.218249 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:21.219885 17381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:21.221474 17381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:21.228132 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.224007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37133 (local address 127.0.0.1:59454)
0924 00:13:21.224455 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.224465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.224532 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.224901 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.224901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.224902 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:21.224969 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.224970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.225842 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.225843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.226831 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.226834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.227800 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.227801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.227887 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.227902 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.227971 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.228036 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260924 00:13:21.228286 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.223813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37615 (local address 127.0.0.1:54800)
0924 00:13:21.223976 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.224015 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.224077 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.224373 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.224376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.224378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:21.224449 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.224450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.225702 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.225703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.227031 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.227034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.228051 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.228052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.228112 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.228116 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.228172 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.228214 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":39,"thread_start_us":59,"threads_started":1}
I20260924 00:13:21.228462 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.224271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.98.1:37133)
0924 00:13:21.224423 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.224428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.224530 (+ 102us) server_negotiation.cc:407] Connection header received
0924 00:13:21.224611 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.224614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.224630 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.224823 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:21.225045 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.225740 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.226896 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.227641 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.227709 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.227765 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.228334 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.228387 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260924 00:13:21.228652 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.224001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.0.98.3:37615)
0924 00:13:21.224227 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.224232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.224243 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:21.224283 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.224285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.224301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.224373 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:21.224504 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.225582 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.227276 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.227966 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.228428 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.228497 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.228523 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.228564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260924 00:13:21.229171 17506 tablet_service.cc:1467] Processing CreateTablet for tablet 8517483c0af9492ea37077eac152ba7b (DEFAULT_TABLE table=test-table [id=4a710ac15518425c9a05e945f60fae57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:21.229369 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.223813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40087 (local address 127.0.0.1:37180)
0924 00:13:21.224188 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.224200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.224243 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.224897 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.224898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.224900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:21.224959 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.224961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.226181 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.226182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.227325 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.227328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.229038 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.229039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.229137 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.229142 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.229240 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.229280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260924 00:13:21.229419 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.223934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.0.98.2:40087)
0924 00:13:21.224665 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.224671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.224680 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:21.224717 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.224719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.224733 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.224805 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:21.225024 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.226070 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.227472 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.228946 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.229025 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.229079 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.229272 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.229324 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":93,"threads_started":1}
I20260924 00:13:21.229539 17506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8517483c0af9492ea37077eac152ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:21.230177 17639 tablet_service.cc:1467] Processing CreateTablet for tablet 8517483c0af9492ea37077eac152ba7b (DEFAULT_TABLE table=test-table [id=4a710ac15518425c9a05e945f60fae57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:21.230496 17639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8517483c0af9492ea37077eac152ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:21.233183 17857 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Bootstrap starting.
I20260924 00:13:21.233610 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 8517483c0af9492ea37077eac152ba7b (DEFAULT_TABLE table=test-table [id=4a710ac15518425c9a05e945f60fae57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:21.233948 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8517483c0af9492ea37077eac152ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:21.234227 17857 tablet_bootstrap.cc:654] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:21.233935 17858 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Bootstrap starting.
I20260924 00:13:21.234854 17857 log.cc:824] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:21.234929 17858 tablet_bootstrap.cc:654] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:21.235761 17858 log.cc:824] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:21.236580 17858 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: No bootstrap required, opened a new log
I20260924 00:13:21.236650 17858 ts_tablet_manager.cc:1397] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:13:21.237023 17861 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Bootstrap starting.
I20260924 00:13:21.237748 17861 tablet_bootstrap.cc:654] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:21.238277 17861 log.cc:824] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:21.239146 17861 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: No bootstrap required, opened a new log
I20260924 00:13:21.239260 17861 ts_tablet_manager.cc:1397] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:21.239884 17857 tablet_bootstrap.cc:492] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: No bootstrap required, opened a new log
I20260924 00:13:21.239773 17858 raft_consensus.cc:348] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.239951 17858 raft_consensus.cc:374] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:21.239961 17857 ts_tablet_manager.cc:1397] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 00:13:21.239998 17858 raft_consensus.cc:729] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc845975e4c9464195a61024cd7e9c26, State: Initialized, Role: FOLLOWER
I20260924 00:13:21.240159 17858 consensus_queue.cc:260] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.240442 17858 ts_tablet_manager.cc:1428] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:13:21.240566 17706 heartbeater.cc:503] Master 127.0.98.62:41643 was elected leader, sending a full tablet report...
I20260924 00:13:21.241045 17861 raft_consensus.cc:348] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.241201 17861 raft_consensus.cc:374] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:21.241235 17861 raft_consensus.cc:729] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96816f783c6b4bc28c997c835a55075e, State: Initialized, Role: FOLLOWER
I20260924 00:13:21.241336 17861 consensus_queue.cc:260] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.241577 17861 ts_tablet_manager.cc:1428] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:21.241694 17840 heartbeater.cc:503] Master 127.0.98.62:41643 was elected leader, sending a full tablet report...
I20260924 00:13:21.242137 17857 raft_consensus.cc:348] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.242377 17857 raft_consensus.cc:374] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:21.242501 17857 raft_consensus.cc:729] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0093518a501244b6b78876a92d7cd702, State: Initialized, Role: FOLLOWER
I20260924 00:13:21.242611 17857 consensus_queue.cc:260] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.242898 17857 ts_tablet_manager.cc:1428] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:21.243044 17572 heartbeater.cc:503] Master 127.0.98.62:41643 was elected leader, sending a full tablet report...
W20260924 00:13:21.277521 17707 tablet.cc:2369] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:13:21.367226 17573 tablet.cc:2369] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:21.391266 17864 raft_consensus.cc:482] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:21.391418 17864 raft_consensus.cc:504] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.391820 17864 leader_election.cc:288] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc845975e4c9464195a61024cd7e9c26 (127.0.98.2:40087), 0093518a501244b6b78876a92d7cd702 (127.0.98.1:37133)
I20260924 00:13:21.394812 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.392238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48141 (local address 127.0.98.1:37133)
0924 00:13:21.392305 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.392307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.392435 (+ 128us) server_negotiation.cc:407] Connection header received
0924 00:13:21.392486 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.392487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.392497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.392553 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:21.392687 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.393160 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394089 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.394454 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394511 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.394520 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.394699 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.394744 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:13:21.394819 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.391995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37133 (local address 127.0.98.3:48141)
0924 00:13:21.392373 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.392381 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.392425 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.392561 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.392562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.392563 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:21.392625 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.392627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.393233 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.393233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.393970 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.393972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.394533 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.394618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.394621 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.394675 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.394709 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260924 00:13:21.394896 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.392103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56127 (local address 127.0.98.2:40087)
0924 00:13:21.392168 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.392170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.392181 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:21.392212 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.392214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.392226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.392291 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:21.392496 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.393057 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394117 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.394482 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.394583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.394780 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.394818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:13:21.394898 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.391983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40087 (local address 127.0.98.3:56127)
0924 00:13:21.392036 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.392050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.392103 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.392311 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.392312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.392314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:21.392395 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.392396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.393133 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.393134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.393953 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.393955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.394585 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.394586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.394694 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.394697 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.394756 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.394791 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260924 00:13:21.395253 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8517483c0af9492ea37077eac152ba7b" candidate_uuid: "96816f783c6b4bc28c997c835a55075e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0093518a501244b6b78876a92d7cd702" is_pre_election: true
I20260924 00:13:21.395267 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8517483c0af9492ea37077eac152ba7b" candidate_uuid: "96816f783c6b4bc28c997c835a55075e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc845975e4c9464195a61024cd7e9c26" is_pre_election: true
I20260924 00:13:21.395421 17526 raft_consensus.cc:2454] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96816f783c6b4bc28c997c835a55075e in term 0.
I20260924 00:13:21.395421 17660 raft_consensus.cc:2454] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96816f783c6b4bc28c997c835a55075e in term 0.
I20260924 00:13:21.395646 17730 leader_election.cc:302] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0093518a501244b6b78876a92d7cd702, 96816f783c6b4bc28c997c835a55075e; no voters:
I20260924 00:13:21.395762 17864 raft_consensus.cc:2781] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:21.395823 17864 raft_consensus.cc:482] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:21.395848 17864 raft_consensus.cc:3037] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:21.396543 17864 raft_consensus.cc:504] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.396688 17864 leader_election.cc:288] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [CANDIDATE]: Term 1 election: Requested vote from peers dc845975e4c9464195a61024cd7e9c26 (127.0.98.2:40087), 0093518a501244b6b78876a92d7cd702 (127.0.98.1:37133)
I20260924 00:13:21.396852 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8517483c0af9492ea37077eac152ba7b" candidate_uuid: "96816f783c6b4bc28c997c835a55075e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc845975e4c9464195a61024cd7e9c26"
I20260924 00:13:21.396867 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8517483c0af9492ea37077eac152ba7b" candidate_uuid: "96816f783c6b4bc28c997c835a55075e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0093518a501244b6b78876a92d7cd702"
I20260924 00:13:21.396937 17526 raft_consensus.cc:3037] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:21.396937 17660 raft_consensus.cc:3037] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:21.397802 17526 raft_consensus.cc:2454] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96816f783c6b4bc28c997c835a55075e in term 1.
I20260924 00:13:21.397802 17660 raft_consensus.cc:2454] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96816f783c6b4bc28c997c835a55075e in term 1.
I20260924 00:13:21.398002 17730 leader_election.cc:302] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0093518a501244b6b78876a92d7cd702, 96816f783c6b4bc28c997c835a55075e; no voters:
I20260924 00:13:21.398098 17864 raft_consensus.cc:2781] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:21.398241 17864 raft_consensus.cc:686] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 1 LEADER]: Becoming Leader. State: Replica: 96816f783c6b4bc28c997c835a55075e, State: Running, Role: LEADER
I20260924 00:13:21.398356 17864 consensus_queue.cc:237] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } }
I20260924 00:13:21.399295 17381 catalog_manager.cc:5495] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e reported cstate change: term changed from 0 to 1, leader changed from <none> to 96816f783c6b4bc28c997c835a55075e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "96816f783c6b4bc28c997c835a55075e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96816f783c6b4bc28c997c835a55075e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 } health_report { overall_health: UNKNOWN } } }
I20260924 00:13:21.417476 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.415137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.0.98.3:37615)
0924 00:13:21.415187 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.415189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.415197 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:21.415331 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.415332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.415347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.415400 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:21.415513 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.415959 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.416684 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.416858 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.416908 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.417041 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:21.417164 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:21.417224 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.417287 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.417303 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:13:21.419278 17526 raft_consensus.cc:1261] T 8517483c0af9492ea37077eac152ba7b P 0093518a501244b6b78876a92d7cd702 [term 1 FOLLOWER]: Refusing update from remote peer 96816f783c6b4bc28c997c835a55075e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:21.419273 17660 raft_consensus.cc:1261] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 1 FOLLOWER]: Refusing update from remote peer 96816f783c6b4bc28c997c835a55075e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:21.419656 17867 consensus_queue.cc:1035] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0093518a501244b6b78876a92d7cd702" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:21.419788 17864 consensus_queue.cc:1035] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc845975e4c9464195a61024cd7e9c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:21.422267 17871 mvcc.cc:205] Tried to move back new op lower bound from 7332695250610536448 to 7332695250528608256. Current Snapshot: MvccSnapshot[applied={T|T < 7332695250610536448}]
I20260924 00:13:21.423273 17872 mvcc.cc:205] Tried to move back new op lower bound from 7332695250610536448 to 7332695250528608256. Current Snapshot: MvccSnapshot[applied={T|T < 7332695250610536448 or (T in {7332695250610536448})}]
I20260924 00:13:21.424645 17380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:21.424742 17380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:21.425526 17380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3503948536b74bde862c49d3940be6f3: Sending DeleteTablet for 3 replicas of tablet 8517483c0af9492ea37077eac152ba7b
I20260924 00:13:21.425894 17639 tablet_service.cc:1514] Processing DeleteTablet for tablet 8517483c0af9492ea37077eac152ba7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37180
I20260924 00:13:21.426142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17443
I20260924 00:13:21.426136 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 8517483c0af9492ea37077eac152ba7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54800
I20260924 00:13:21.427603 17880 tablet_replica.cc:330] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: stopping tablet replica
I20260924 00:13:21.427716 17880 raft_consensus.cc:2229] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:21.427816 17880 raft_consensus.cc:2258] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:21.428287 17880 ts_tablet_manager.cc:1905] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:21.430408 17880 ts_tablet_manager.cc:1918] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:21.430490 17880 log.cc:1196] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/8517483c0af9492ea37077eac152ba7b
I20260924 00:13:21.430764 17880 ts_tablet_manager.cc:1939] T 8517483c0af9492ea37077eac152ba7b P 96816f783c6b4bc28c997c835a55075e: Deleting consensus metadata
I20260924 00:13:21.431398 17368 catalog_manager.cc:4841] TS 96816f783c6b4bc28c997c835a55075e (127.0.98.3:37615): tablet 8517483c0af9492ea37077eac152ba7b (table test-table [id=4a710ac15518425c9a05e945f60fae57]) successfully deleted
I20260924 00:13:21.431461 17879 tablet_replica.cc:330] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: stopping tablet replica
I20260924 00:13:21.431586 17879 raft_consensus.cc:2229] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:21.431717 17879 raft_consensus.cc:2258] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:21.432413 17879 ts_tablet_manager.cc:1905] T 8517483c0af9492ea37077eac152ba7b P dc845975e4c9464195a61024cd7e9c26: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:13:21.432586 17368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37133 (error 108)
I20260924 00:13:21.432834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17577
I20260924 00:13:21.433923 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.433069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37133 (local address 127.0.0.1:59468)
0924 00:13:21.433860 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.433896 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":769}
W20260924 00:13:21.435560 17368 catalog_manager.cc:4584] TS 0093518a501244b6b78876a92d7cd702 (127.0.98.1:37133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8517483c0af9492ea37077eac152ba7b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37133: connect: Connection refused (error 111)
I20260924 00:13:21.438825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17711
I20260924 00:13:21.438920 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.438795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40087 (local address 127.0.0.1:37188)
0924 00:13:21.438855 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.438883 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:13:21.443547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17349
2026-09-24T00:13:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T00:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:21Z Disabled control of system clock
I20260924 00:13:21.471379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:21.552680 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:21.552922 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:21.552945 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:21.554088 17885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:21.554127 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:21.554140 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:21.554152 17885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:21.554164 17885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:21.555502 17885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:21.555701 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:21.555963 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:21.560966 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.560962 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.561038 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.561218 17885 server_base.cc:1034] running on GCE node
I20260924 00:13:21.561580 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:21.561847 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:21.562992 17885 hybrid_clock.cc:648] HybridClock initialized: now 1790208801562977 us; error 32 us; skew 500 ppm
I20260924 00:13:21.564474 17885 webserver.cc:460] Webserver started at http://127.0.98.62:46593/ using document root <none> and password file <none>
I20260924 00:13:21.564759 17885 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:21.564810 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:21.564985 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:21.566057 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "79a285fe9bdf422d8e803a74a3d3469c"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "372bbd812d62a300ac96a9a186fe7030"
server_key_iv: "766be612be49677604ca7346280bb14c"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.566419 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "79a285fe9bdf422d8e803a74a3d3469c"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "372bbd812d62a300ac96a9a186fe7030"
server_key_iv: "766be612be49677604ca7346280bb14c"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.567926 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:21.568814 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.569144 17885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:21.569275 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "79a285fe9bdf422d8e803a74a3d3469c"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "372bbd812d62a300ac96a9a186fe7030"
server_key_iv: "766be612be49677604ca7346280bb14c"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.569370 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.583527 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.583869 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.584023 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:21.587990 17885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42353
I20260924 00:13:21.588028 17951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42353 every 8 connection(s)
I20260924 00:13:21.588383 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:21.589340 17952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:21.591708 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: Bootstrap starting.
I20260924 00:13:21.592320 17952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:21.592669 17952 log.cc:824] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:21.593376 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: No bootstrap required, opened a new log
I20260924 00:13:21.594961 17952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260924 00:13:21.595103 17952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:21.595126 17952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a285fe9bdf422d8e803a74a3d3469c, State: Initialized, Role: FOLLOWER
I20260924 00:13:21.595312 17952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260924 00:13:21.595382 17952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:21.595405 17952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:21.595474 17952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:21.595816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17885
I20260924 00:13:21.595913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:21.596112 392 external_mini_cluster.cc:1428] Setting key 1d0197ab0748892a86bc838bacd45a1a
I20260924 00:13:21.596252 17952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260924 00:13:21.596359 17952 leader_election.cc:302] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a285fe9bdf422d8e803a74a3d3469c; no voters:
I20260924 00:13:21.596522 17952 leader_election.cc:288] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:21.596589 17955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:21.597136 17952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:21.597280 17955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [term 1 LEADER]: Becoming Leader. State: Replica: 79a285fe9bdf422d8e803a74a3d3469c, State: Running, Role: LEADER
I20260924 00:13:21.597433 17955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } }
I20260924 00:13:21.597846 17958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a285fe9bdf422d8e803a74a3d3469c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } }
I20260924 00:13:21.597855 17959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a285fe9bdf422d8e803a74a3d3469c. Latest consensus state: current_term: 1 leader_uuid: "79a285fe9bdf422d8e803a74a3d3469c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a285fe9bdf422d8e803a74a3d3469c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42353 } } }
I20260924 00:13:21.597967 17958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:21.597977 17959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:21.598169 17966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:21.598649 17966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:21.600669 17966 catalog_manager.cc:1347] Generated new cluster ID: 6f71fb508a6342e6bbfdebaaf76d56ac
I20260924 00:13:21.600726 17966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:21.602825 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.596639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.0.98.62:42353)
0924 00:13:21.596916 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.596926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.596963 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:13:21.597091 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.597099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.597180 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.597607 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.597705 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.599475 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.600618 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.601221 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.601336 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.602113 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.602116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.602117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:21.602315 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.602352 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:21.602365 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.602424 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.602531 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.602602 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":159,"threads_started":1}
I20260924 00:13:21.617432 17966 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:21.618023 17966 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:21.629830 17966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: Generated new TSK 0
I20260924 00:13:21.630173 17966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:21.633005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:21.715627 17976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:21.715842 17976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:21.715865 17976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:21.717067 17976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:21.717113 17976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:21.717139 17976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:21.718552 17976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:21.718806 17976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:21.719156 17976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:21.725135 17981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.725294 17976 server_base.cc:1034] running on GCE node
W20260924 00:13:21.725116 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.725136 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.725677 17976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:21.725942 17976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:21.727128 17976 hybrid_clock.cc:648] HybridClock initialized: now 1790208801727103 us; error 40 us; skew 500 ppm
I20260924 00:13:21.728449 17976 webserver.cc:460] Webserver started at http://127.0.98.1:45425/ using document root <none> and password file <none>
I20260924 00:13:21.728713 17976 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:21.728787 17976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:21.728933 17976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:21.729990 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "d511211a34004108a71c6433713acfa4"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "4c6980660e7b65e7290081927a38ce4d"
server_key_iv: "50cc032faad53ea8414d2b33fa5373b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.730391 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d511211a34004108a71c6433713acfa4"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "4c6980660e7b65e7290081927a38ce4d"
server_key_iv: "50cc032faad53ea8414d2b33fa5373b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.731781 17976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:21.732564 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.732745 17976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.732842 17976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "d511211a34004108a71c6433713acfa4"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "4c6980660e7b65e7290081927a38ce4d"
server_key_iv: "50cc032faad53ea8414d2b33fa5373b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.732921 17976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.750231 17976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.750574 17976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.750737 17976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:21.750969 17976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:21.751370 17976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:21.751428 17976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.751466 17976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:21.751492 17976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.757077 17976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45179
I20260924 00:13:21.757165 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45179 every 8 connection(s)
I20260924 00:13:21.757511 17976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:13:21.757972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 17976
I20260924 00:13:21.758075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:21.758272 392 external_mini_cluster.cc:1428] Setting key 6643aa4c24514fcd032aabb85012e467
I20260924 00:13:21.759469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:13:21.764062 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.758386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42799 (local address 127.0.98.62:42353)
0924 00:13:21.758448 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.758450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.758763 (+ 313us) server_negotiation.cc:407] Connection header received
0924 00:13:21.759490 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.759491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.759503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.759587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.760465 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.760963 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.762543 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.762753 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.762813 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.763812 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.763813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.763813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:21.763818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.763830 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:21.763834 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.763876 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.763971 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.764001 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:13:21.764379 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.758397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.98.1:42799)
0924 00:13:21.758617 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.758663 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.759382 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.759658 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.759662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.759672 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:21.760369 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.760375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.761076 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.761078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.762444 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.762447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.762844 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.762845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.763024 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.763640 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:21.763641 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:21.763684 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:21.763718 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:21.763890 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:21.763894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:21.763898 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.763971 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.764103 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260924 00:13:21.765024 18104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42353
I20260924 00:13:21.765142 18104 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:21.765358 18104 heartbeater.cc:511] Master 127.0.98.62:42353 requested a full tablet report, sending...
I20260924 00:13:21.765897 17916 ts_manager.cc:194] Registered new tserver with Master: d511211a34004108a71c6433713acfa4 (127.0.98.1:45179)
I20260924 00:13:21.766546 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42799
W20260924 00:13:21.842459 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:21.842677 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:21.842700 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:21.843923 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:21.843971 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:21.843997 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:21.845738 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:21.846007 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:21.846354 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:21.851507 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.851534 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.851509 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.851787 18107 server_base.cc:1034] running on GCE node
I20260924 00:13:21.852170 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:21.852437 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:21.853672 18107 hybrid_clock.cc:648] HybridClock initialized: now 1790208801853669 us; error 43 us; skew 500 ppm
I20260924 00:13:21.855136 18107 webserver.cc:460] Webserver started at http://127.0.98.2:44055/ using document root <none> and password file <none>
I20260924 00:13:21.855413 18107 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:21.855486 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:21.855654 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:21.856788 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "c2ac9228406849118f1948f2ee24239d"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "e807da987158ce66d69e18c06cfc16ce"
server_key_iv: "c5cf5811fd391e8159c5757587114279"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.857157 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c2ac9228406849118f1948f2ee24239d"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "e807da987158ce66d69e18c06cfc16ce"
server_key_iv: "c5cf5811fd391e8159c5757587114279"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.858443 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:21.859166 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.859361 18107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:21.859460 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "c2ac9228406849118f1948f2ee24239d"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "e807da987158ce66d69e18c06cfc16ce"
server_key_iv: "c5cf5811fd391e8159c5757587114279"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.859549 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.874012 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.874322 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.874480 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:21.874706 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:21.875047 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:21.875106 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.875142 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:21.875168 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.881222 18107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45079
I20260924 00:13:21.881269 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45079 every 8 connection(s)
I20260924 00:13:21.881608 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:21.885221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18107
I20260924 00:13:21.885324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:21.885535 392 external_mini_cluster.cc:1428] Setting key c22df0b25b72e44cfcb432ea46d63ce4
I20260924 00:13:21.886744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--builtin_ntp_servers=127.0.98.20:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:13:21.887750 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.882345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43647 (local address 127.0.98.62:42353)
0924 00:13:21.882399 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:21.882401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:21.882696 (+ 295us) server_negotiation.cc:407] Connection header received
0924 00:13:21.883197 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:21.883198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:21.883209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:21.883300 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:21.884012 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.884564 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.885961 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.886322 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.886402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.887391 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:21.887392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:21.887393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:21.887399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:21.887416 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:21.887422 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:21.887487 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:21.887581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 00:13:21.887619 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:21.887867 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:21.882328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.98.2:43647)
0924 00:13:21.882558 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:21.882598 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:21.883075 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:21.883379 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:21.883383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:21.883391 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:21.883913 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.883918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.884692 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.884693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.885887 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:21.885890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:21.886419 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:21.886420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:21.886577 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:21.887201 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:21.887202 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:21.887268 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:21.887300 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:21.887494 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:21.887498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:21.887501 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:21.887553 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:13:21.887669 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260924 00:13:21.888386 18235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42353
I20260924 00:13:21.888499 18235 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:21.888690 18235 heartbeater.cc:511] Master 127.0.98.62:42353 requested a full tablet report, sending...
I20260924 00:13:21.889046 17916 ts_manager.cc:194] Registered new tserver with Master: c2ac9228406849118f1948f2ee24239d (127.0.98.2:45079)
I20260924 00:13:21.889379 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43647
W20260924 00:13:21.969811 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:21.970026 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:21.970048 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:21.971123 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:21.971160 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:21.971184 18238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:21.972815 18238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:21.973044 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:21.973304 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:21.978332 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.978364 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:21.978338 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:21.978475 18238 server_base.cc:1034] running on GCE node
I20260924 00:13:21.978747 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:21.978958 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:21.980263 18238 hybrid_clock.cc:648] HybridClock initialized: now 1790208801980246 us; error 37 us; skew 500 ppm
I20260924 00:13:21.981566 18238 webserver.cc:460] Webserver started at http://127.0.98.3:41971/ using document root <none> and password file <none>
I20260924 00:13:21.981835 18238 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:21.981909 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:21.982059 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:21.983180 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "83967263873141629cf6d62ceb80b5a5"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "c628cbdb863560b3c952f4783a2a00d3"
server_key_iv: "ddaf70b2589d77ff3d5dd23047c44552"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.983587 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83967263873141629cf6d62ceb80b5a5"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "c628cbdb863560b3c952f4783a2a00d3"
server_key_iv: "ddaf70b2589d77ff3d5dd23047c44552"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.985126 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:21.985989 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:21.986255 18238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:21.986359 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "83967263873141629cf6d62ceb80b5a5"
format_stamp: "Formatted at 2026-09-24 00:13:21 on dist-test-slave-3q8k"
server_key: "c628cbdb863560b3c952f4783a2a00d3"
server_key_iv: "ddaf70b2589d77ff3d5dd23047c44552"
server_key_version: "encryptionkey@0"
I20260924 00:13:21.986455 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:21.999413 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:21.999737 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:21.999895 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:22.000131 18238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:22.000464 18238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:22.000528 18238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:22.000573 18238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:22.000599 18238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:22.006331 18238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40749
I20260924 00:13:22.006435 18365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40749 every 8 connection(s)
I20260924 00:13:22.006757 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:13:22.012416 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.007568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42321 (local address 127.0.98.62:42353)
0924 00:13:22.007638 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.007640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.007911 (+ 271us) server_negotiation.cc:407] Connection header received
0924 00:13:22.008530 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.008532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.008545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.008600 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:22.009298 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.009821 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.010839 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.011067 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.011149 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.012091 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:22.012093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:22.012093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:22.012099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:22.012117 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:22.012122 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:22.012181 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.012288 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.012324 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:13:22.012650 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.007554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42353 (local address 127.0.98.3:42321)
0924 00:13:22.007778 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.007821 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.008400 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.008681 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.008692 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.008701 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:22.009201 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.009205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.009936 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.009937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.010739 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.010741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.011159 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.011160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.011339 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.011868 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:22.011869 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:22.011903 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:22.011929 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:22.012174 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:22.012180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:22.012184 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.012293 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.012406 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260924 00:13:22.012845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18238
I20260924 00:13:22.012940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:22.013121 18366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42353
I20260924 00:13:22.013160 392 external_mini_cluster.cc:1428] Setting key ec02e1f1ac1f4a99e378de5210002af9
I20260924 00:13:22.013208 18366 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:22.013429 18366 heartbeater.cc:511] Master 127.0.98.62:42353 requested a full tablet report, sending...
I20260924 00:13:22.013868 17916 ts_manager.cc:194] Registered new tserver with Master: 83967263873141629cf6d62ceb80b5a5 (127.0.98.3:40749)
I20260924 00:13:22.014348 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42321
I20260924 00:13:22.014437 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:22.018838 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.015771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.98.62:42353)
0924 00:13:22.015848 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.015850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.015941 (+ 91us) server_negotiation.cc:407] Connection header received
0924 00:13:22.016025 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.016026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.016039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.016084 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:22.016313 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.016803 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.017585 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.017792 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.017879 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.018582 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:22.018584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:22.018584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:22.018589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:22.018602 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:22.018606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:22.018664 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.018729 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.018752 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:13:22.020599 17916 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:22.022169 17916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:22.029912 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45079 (local address 127.0.0.1:49356)
0924 00:13:22.025414 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.025423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.025473 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.025687 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.025690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.025692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:22.025762 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.025764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.027292 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.027293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.028321 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.028323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.029431 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.029432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.029661 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.029687 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.029763 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.029825 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":136,"thread_start_us":100,"threads_started":1}
I20260924 00:13:22.029942 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.0.98.2:45079)
0924 00:13:22.025086 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.025092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.025529 (+ 437us) server_negotiation.cc:407] Connection header received
0924 00:13:22.025577 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.025579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.025591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.025676 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:22.025859 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.027119 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.028422 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.029232 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.029305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.029397 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.029818 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.029857 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 00:13:22.030915 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40749 (local address 127.0.0.1:51174)
0924 00:13:22.024808 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.024819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.024905 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.025965 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.025966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.025968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:22.026003 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.026004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.028285 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.028286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.029635 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.029639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.030649 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.030650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.030714 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.030718 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.030791 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.030836 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260924 00:13:22.030928 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.0.98.3:40749)
0924 00:13:22.025804 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.025809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.025818 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:22.025855 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.025857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.025870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.025969 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:22.026067 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.026811 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.029746 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.030539 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.030630 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.030715 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.030806 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.030849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":76,"threads_started":1}
I20260924 00:13:22.030805 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad (DEFAULT_TABLE table=test-table [id=ea6f6b6f87af42e3a349fb27822bb44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:22.031260 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:22.031529 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.0.98.1:45179)
0924 00:13:22.025136 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.025140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.025149 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:22.025220 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.025222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.025238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.025313 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:22.026889 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.028175 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.030142 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.030950 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.031001 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.031096 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.031361 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.031417 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":70,"threads_started":1}
I20260924 00:13:22.031773 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.024562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45179 (local address 127.0.0.1:53054)
0924 00:13:22.024712 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.024755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.025071 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.026761 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.026762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.026765 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:22.026810 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.026811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.028273 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.028274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.030038 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.030041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.031152 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.031152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.031200 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.031202 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.031629 (+ 427us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.031683 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":41,"thread_start_us":80,"threads_started":1}
I20260924 00:13:22.031699 18300 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad (DEFAULT_TABLE table=test-table [id=ea6f6b6f87af42e3a349fb27822bb44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:22.032051 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:22.034865 18380 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Bootstrap starting.
I20260924 00:13:22.035511 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad (DEFAULT_TABLE table=test-table [id=ea6f6b6f87af42e3a349fb27822bb44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:22.035895 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:22.036206 18380 tablet_bootstrap.cc:654] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:22.036576 18380 log.cc:824] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:22.037371 18381 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Bootstrap starting.
I20260924 00:13:22.037453 18380 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: No bootstrap required, opened a new log
I20260924 00:13:22.037531 18380 ts_tablet_manager.cc:1397] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:13:22.038071 18381 tablet_bootstrap.cc:654] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:22.038460 18381 log.cc:824] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:22.039325 18384 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Bootstrap starting.
I20260924 00:13:22.039367 18381 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: No bootstrap required, opened a new log
I20260924 00:13:22.039430 18381 ts_tablet_manager.cc:1397] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:22.040022 18384 tablet_bootstrap.cc:654] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:22.040411 18384 log.cc:824] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:22.040598 18380 raft_consensus.cc:348] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.040796 18380 raft_consensus.cc:374] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:22.040836 18380 raft_consensus.cc:729] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ac9228406849118f1948f2ee24239d, State: Initialized, Role: FOLLOWER
I20260924 00:13:22.040992 18380 consensus_queue.cc:260] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.041217 18384 tablet_bootstrap.cc:492] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: No bootstrap required, opened a new log
I20260924 00:13:22.041286 18384 ts_tablet_manager.cc:1397] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:13:22.041289 18380 ts_tablet_manager.cc:1428] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:13:22.041420 18235 heartbeater.cc:503] Master 127.0.98.62:42353 was elected leader, sending a full tablet report...
I20260924 00:13:22.042335 18381 raft_consensus.cc:348] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.042539 18381 raft_consensus.cc:374] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:22.042599 18381 raft_consensus.cc:729] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83967263873141629cf6d62ceb80b5a5, State: Initialized, Role: FOLLOWER
I20260924 00:13:22.042732 18381 consensus_queue.cc:260] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.043042 18381 ts_tablet_manager.cc:1428] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:13:22.043058 18366 heartbeater.cc:503] Master 127.0.98.62:42353 was elected leader, sending a full tablet report...
I20260924 00:13:22.043995 18384 raft_consensus.cc:348] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.044126 18384 raft_consensus.cc:374] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:22.044148 18384 raft_consensus.cc:729] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d511211a34004108a71c6433713acfa4, State: Initialized, Role: FOLLOWER
I20260924 00:13:22.044298 18384 consensus_queue.cc:260] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.044546 18384 ts_tablet_manager.cc:1428] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 00:13:22.044621 18104 heartbeater.cc:503] Master 127.0.98.62:42353 was elected leader, sending a full tablet report...
I20260924 00:13:22.084460 18387 raft_consensus.cc:482] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:22.084599 18387 raft_consensus.cc:504] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.085021 18387 leader_election.cc:288] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d511211a34004108a71c6433713acfa4 (127.0.98.1:45179), c2ac9228406849118f1948f2ee24239d (127.0.98.2:45079)
I20260924 00:13:22.087939 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.085420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39455 (local address 127.0.98.1:45179)
0924 00:13:22.085478 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.085481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.085489 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:22.085517 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.085518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.085530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.085603 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:22.085792 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.086268 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.087191 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.087582 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.087674 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.087684 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.087833 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.087874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:13:22.087942 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.085265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45179 (local address 127.0.98.3:39455)
0924 00:13:22.085319 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.085333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.085377 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.085655 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.085655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.085657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:22.085714 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.085716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.086350 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.086351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.087098 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.087100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.087677 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.087678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.087747 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.087750 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.087825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.087868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":6}
I20260924 00:13:22.088385 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.085390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45079 (local address 127.0.98.3:50359)
0924 00:13:22.085638 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.085647 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:22.085737 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:22.085899 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:22.085900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:22.085902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:22.085980 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.085982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.086667 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.086669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.087492 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:22.087494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.088112 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.088113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:22.088201 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.088204 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:22.088267 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:13:22.088302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":164,"threads_started":1}
I20260924 00:13:22.088327 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6fe49e4df64a64ad815c94c8cbc4ad" candidate_uuid: "83967263873141629cf6d62ceb80b5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d511211a34004108a71c6433713acfa4" is_pre_election: true
I20260924 00:13:22.088398 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.085497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50359 (local address 127.0.98.2:45079)
0924 00:13:22.085582 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.085584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.085720 (+ 136us) server_negotiation.cc:407] Connection header received
0924 00:13:22.085798 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.085799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.085808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.085892 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:22.086119 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.086586 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.087588 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.087947 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.088057 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.088068 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.088279 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.088318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:13:22.088490 18058 raft_consensus.cc:2454] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83967263873141629cf6d62ceb80b5a5 in term 0.
I20260924 00:13:22.088732 18255 leader_election.cc:302] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83967263873141629cf6d62ceb80b5a5, d511211a34004108a71c6433713acfa4; no voters:
I20260924 00:13:22.088851 18387 raft_consensus.cc:2781] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:22.088843 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6fe49e4df64a64ad815c94c8cbc4ad" candidate_uuid: "83967263873141629cf6d62ceb80b5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac9228406849118f1948f2ee24239d" is_pre_election: true
I20260924 00:13:22.088908 18387 raft_consensus.cc:482] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:22.088943 18387 raft_consensus.cc:3037] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:22.088981 18189 raft_consensus.cc:2454] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83967263873141629cf6d62ceb80b5a5 in term 0.
I20260924 00:13:22.089778 18387 raft_consensus.cc:504] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.089905 18387 leader_election.cc:288] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [CANDIDATE]: Term 1 election: Requested vote from peers d511211a34004108a71c6433713acfa4 (127.0.98.1:45179), c2ac9228406849118f1948f2ee24239d (127.0.98.2:45079)
I20260924 00:13:22.090080 18189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6fe49e4df64a64ad815c94c8cbc4ad" candidate_uuid: "83967263873141629cf6d62ceb80b5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ac9228406849118f1948f2ee24239d"
I20260924 00:13:22.090096 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6fe49e4df64a64ad815c94c8cbc4ad" candidate_uuid: "83967263873141629cf6d62ceb80b5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d511211a34004108a71c6433713acfa4"
I20260924 00:13:22.090145 18189 raft_consensus.cc:3037] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:22.090206 18058 raft_consensus.cc:3037] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:22.090975 18189 raft_consensus.cc:2454] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83967263873141629cf6d62ceb80b5a5 in term 1.
I20260924 00:13:22.091053 18058 raft_consensus.cc:2454] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83967263873141629cf6d62ceb80b5a5 in term 1.
I20260924 00:13:22.091174 18255 leader_election.cc:302] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83967263873141629cf6d62ceb80b5a5, c2ac9228406849118f1948f2ee24239d; no voters:
I20260924 00:13:22.091271 18387 raft_consensus.cc:2781] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:22.091429 18387 raft_consensus.cc:686] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 83967263873141629cf6d62ceb80b5a5, State: Running, Role: LEADER
I20260924 00:13:22.091522 18387 consensus_queue.cc:237] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } }
I20260924 00:13:22.092353 17916 catalog_manager.cc:5495] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83967263873141629cf6d62ceb80b5a5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "83967263873141629cf6d62ceb80b5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83967263873141629cf6d62ceb80b5a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 } health_report { overall_health: UNKNOWN } } }
I20260924 00:13:22.101751 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.099364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.3:40749)
0924 00:13:22.099405 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:22.099407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:22.099459 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:13:22.099551 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:22.099554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:22.099574 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:22.099649 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:22.099759 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.100213 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.100960 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:22.101153 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:22.101228 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:22.101406 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:22.101497 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:22.101603 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:22.101627 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 00:13:22.101643 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:13:22.103509 18189 raft_consensus.cc:1261] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 1 FOLLOWER]: Refusing update from remote peer 83967263873141629cf6d62ceb80b5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:22.103547 18058 raft_consensus.cc:1261] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 1 FOLLOWER]: Refusing update from remote peer 83967263873141629cf6d62ceb80b5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:22.103843 18387 consensus_queue.cc:1035] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ac9228406849118f1948f2ee24239d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:22.103971 18390 consensus_queue.cc:1035] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d511211a34004108a71c6433713acfa4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:22.106295 18393 mvcc.cc:205] Tried to move back new op lower bound from 7332695253413310464 to 7332695253367554048. Current Snapshot: MvccSnapshot[applied={T|T < 7332695253413310464}]
I20260924 00:13:22.106673 18394 mvcc.cc:205] Tried to move back new op lower bound from 7332695253413310464 to 7332695253367554048. Current Snapshot: MvccSnapshot[applied={T|T < 7332695253413310464}]
I20260924 00:13:22.106810 18395 mvcc.cc:205] Tried to move back new op lower bound from 7332695253413310464 to 7332695253367554048. Current Snapshot: MvccSnapshot[applied={T|T < 7332695253413310464}]
I20260924 00:13:22.109191 17916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:22.109292 17916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:22.110075 17916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a285fe9bdf422d8e803a74a3d3469c: Sending DeleteTablet for 3 replicas of tablet 9c6fe49e4df64a64ad815c94c8cbc4ad
I20260924 00:13:22.110417 18169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:22 UTC) from {username='slave'} at 127.0.0.1:49356
I20260924 00:13:22.110451 18300 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:22 UTC) from {username='slave'} at 127.0.0.1:51174
I20260924 00:13:22.110451 18038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:22 UTC) from {username='slave'} at 127.0.0.1:53054
I20260924 00:13:22.110680 18404 tablet_replica.cc:330] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: stopping tablet replica
I20260924 00:13:22.110716 18403 tablet_replica.cc:330] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: stopping tablet replica
I20260924 00:13:22.110780 18404 raft_consensus.cc:2229] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:22.110889 18404 raft_consensus.cc:2258] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:22.110939 18403 raft_consensus.cc:2229] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:22.111042 18403 raft_consensus.cc:2258] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:22.111145 18405 tablet_replica.cc:330] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: stopping tablet replica
I20260924 00:13:22.111195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17976
I20260924 00:13:22.111752 18404 ts_tablet_manager.cc:1905] T 9c6fe49e4df64a64ad815c94c8cbc4ad P d511211a34004108a71c6433713acfa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:22.111804 18403 ts_tablet_manager.cc:1905] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:22.111261 18405 raft_consensus.cc:2229] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:22.112200 18405 raft_consensus.cc:2258] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:22.112636 18405 ts_tablet_manager.cc:1905] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:22.113889 18403 ts_tablet_manager.cc:1918] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:22.113952 18403 log.cc:1196] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/9c6fe49e4df64a64ad815c94c8cbc4ad
I20260924 00:13:22.114202 18403 ts_tablet_manager.cc:1939] T 9c6fe49e4df64a64ad815c94c8cbc4ad P c2ac9228406849118f1948f2ee24239d: Deleting consensus metadata
I20260924 00:13:22.114614 17902 catalog_manager.cc:4841] TS c2ac9228406849118f1948f2ee24239d (127.0.98.2:45079): tablet 9c6fe49e4df64a64ad815c94c8cbc4ad (table test-table [id=ea6f6b6f87af42e3a349fb27822bb44a]) successfully deleted
I20260924 00:13:22.114673 18405 ts_tablet_manager.cc:1918] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:22.114738 18405 log.cc:1196] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/9c6fe49e4df64a64ad815c94c8cbc4ad
I20260924 00:13:22.114984 18405 ts_tablet_manager.cc:1939] T 9c6fe49e4df64a64ad815c94c8cbc4ad P 83967263873141629cf6d62ceb80b5a5: Deleting consensus metadata
I20260924 00:13:22.115422 17900 catalog_manager.cc:4841] TS 83967263873141629cf6d62ceb80b5a5 (127.0.98.3:40749): tablet 9c6fe49e4df64a64ad815c94c8cbc4ad (table test-table [id=ea6f6b6f87af42e3a349fb27822bb44a]) successfully deleted
W20260924 00:13:22.118132 17902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45179 (error 108)
I20260924 00:13:22.118372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18107
I20260924 00:13:22.118939 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:22.118475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45179 (local address 127.0.0.1:53064)
0924 00:13:22.118858 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:22.118905 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
W20260924 00:13:22.119572 17902 catalog_manager.cc:4584] TS d511211a34004108a71c6433713acfa4 (127.0.98.1:45179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c6fe49e4df64a64ad815c94c8cbc4ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45179: connect: Connection refused (error 111)
I20260924 00:13:22.123386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18238
I20260924 00:13:22.127413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 17885
2026-09-24T00:13:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (682 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:22 dist-test-slave-3q8k krb5kdc[18409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:22 dist-test-slave-3q8k krb5kdc[18409](info): set up 2 sockets
Sep 24 00:13:22 dist-test-slave-3q8k krb5kdc[18409](info): commencing operation
krb5kdc: starting...
W20260924 00:13:23.159142 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:23.282563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:23.366343 18422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:23.366611 18422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:23.366652 18422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:23.368687 18422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:23.368759 18422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:23.368785 18422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:23.368806 18422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:23.368835 18422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:23.371656 18422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:23.372007 18422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:23.372298 18422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:23.378687 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:23.378847 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:23.378690 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.379541 18422 server_base.cc:1034] running on GCE node
I20260924 00:13:23.379839 18422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:23.380218 18422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:23.381408 18422 hybrid_clock.cc:648] HybridClock initialized: now 1790208803381374 us; error 49 us; skew 500 ppm
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:23.384650 18422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:23.385222 18422 webserver.cc:460] Webserver started at http://127.0.98.62:36775/ using document root <none> and password file <none>
I20260924 00:13:23.385501 18422 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:23.385563 18422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:23.385704 18422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:23.387297 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "de6860fe97764bf5afbabee0802caf86"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "d13d849f95d55246f2e59456ab71c455"
server_key_iv: "552ba516b704a04401cbbbde7fc44622"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.387744 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "de6860fe97764bf5afbabee0802caf86"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "d13d849f95d55246f2e59456ab71c455"
server_key_iv: "552ba516b704a04401cbbbde7fc44622"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.389493 18422 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260924 00:13:23.390422 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.390625 18422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:23.390735 18422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "de6860fe97764bf5afbabee0802caf86"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "d13d849f95d55246f2e59456ab71c455"
server_key_iv: "552ba516b704a04401cbbbde7fc44622"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.390817 18422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:23.408973 18422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:23.411765 18422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:23.411934 18422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:23.417196 18422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40421
I20260924 00:13:23.417224 18489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40421 every 8 connection(s)
I20260924 00:13:23.417671 18422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:23.417948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18422
I20260924 00:13:23.418073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:23.418359 392 external_mini_cluster.cc:1428] Setting key fb17aeb5bfff786cd8cfbe7c815bee7f
I20260924 00:13:23.418860 18490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:23.422255 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: Bootstrap starting.
I20260924 00:13:23.422963 18490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:23.423323 18490 log.cc:824] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:23.424160 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: No bootstrap required, opened a new log
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:23.426980 18490 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } }
I20260924 00:13:23.427152 18490 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:23.427201 18490 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6860fe97764bf5afbabee0802caf86, State: Initialized, Role: FOLLOWER
I20260924 00:13:23.427376 18490 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } }
I20260924 00:13:23.427462 18490 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:23.427507 18490 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:23.427551 18490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:23.428308 18490 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } }
I20260924 00:13:23.428426 18490 leader_election.cc:302] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6860fe97764bf5afbabee0802caf86; no voters:
I20260924 00:13:23.428596 18490 leader_election.cc:288] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:23.428848 18495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:23.428897 18490 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:23.429003 18495 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [term 1 LEADER]: Becoming Leader. State: Replica: de6860fe97764bf5afbabee0802caf86, State: Running, Role: LEADER
I20260924 00:13:23.429081 18495 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } }
I20260924 00:13:23.429477 18495 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6860fe97764bf5afbabee0802caf86. Latest consensus state: current_term: 1 leader_uuid: "de6860fe97764bf5afbabee0802caf86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } } }
I20260924 00:13:23.429484 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de6860fe97764bf5afbabee0802caf86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6860fe97764bf5afbabee0802caf86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40421 } } }
I20260924 00:13:23.429565 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:23.429558 18495 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86 [sys.catalog]: This master's current role is: LEADER
W20260924 00:13:23.430770 18510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:23.430838 18510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:23.432081 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.418901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.0.98.62:40421)
0924 00:13:23.419078 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.419095 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.419192 (+ 97us) server_negotiation.cc:407] Connection header received
0924 00:13:23.420204 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.420213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.420278 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.420426 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:23.421207 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.422628 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.423786 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.424415 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.424529 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.427670 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:23.427675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:23.427675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:23.427686 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:23.429789 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.430405 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.430409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.430409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.430517 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.430768 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.430769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.430770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.431122 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:23.431717 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.431785 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.431886 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260924 00:13:23.435384 18504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:23.435601 18504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:23.439335 18504 catalog_manager.cc:1347] Generated new cluster ID: 69de639e929042cfaa34d0d71d2e5856
I20260924 00:13:23.439396 18504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:23.459053 18504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:23.459687 18504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:23.464854 18504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: Generated new TSK 0
I20260924 00:13:23.465065 18504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:13:23.502223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:23.584791 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:23.585004 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:23.585028 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:23.586196 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:23.586241 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:23.586297 18516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:23.587783 18516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:23.588102 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:23.588343 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:23.593763 18524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.593791 18516 server_base.cc:1034] running on GCE node
W20260924 00:13:23.593763 18521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:23.594056 18522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.594290 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:23.594491 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:23.595628 18516 hybrid_clock.cc:648] HybridClock initialized: now 1790208803595593 us; error 52 us; skew 500 ppm
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:23.598150 18516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:13:23.598528 18516 webserver.cc:460] Webserver started at http://127.0.98.1:39167/ using document root <none> and password file <none>
I20260924 00:13:23.598775 18516 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:23.598821 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:23.598973 18516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:23.600082 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "4376db40709e4c4fb20b7a1806a29bfd"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f56795e983abf4019ede895c9d73c4d0"
server_key_iv: "307bef7b68cad86591d468241f0e5c5f"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.600436 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4376db40709e4c4fb20b7a1806a29bfd"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f56795e983abf4019ede895c9d73c4d0"
server_key_iv: "307bef7b68cad86591d468241f0e5c5f"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.601835 18516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:13:23.602691 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.602882 18516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:23.602955 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "4376db40709e4c4fb20b7a1806a29bfd"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f56795e983abf4019ede895c9d73c4d0"
server_key_iv: "307bef7b68cad86591d468241f0e5c5f"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.603047 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:23.611100 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:23.613149 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:23.613271 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:23.613433 18516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:23.613747 18516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:23.613780 18516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.613803 18516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:23.613842 18516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.619609 18516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39151
I20260924 00:13:23.619696 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39151 every 8 connection(s)
I20260924 00:13:23.620004 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:23.627470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18516
I20260924 00:13:23.627554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:23.627758 392 external_mini_cluster.cc:1428] Setting key df4dbfc3a981de2bb4f4a376b759eefa
I20260924 00:13:23.629671 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.620771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55181 (local address 127.0.98.62:40421)
0924 00:13:23.620838 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.620840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.621029 (+ 189us) server_negotiation.cc:407] Connection header received
0924 00:13:23.621953 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.621954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.621965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.622033 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:23.622620 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.623134 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.624155 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.624349 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.624407 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.626423 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:23.626425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:23.626425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:23.626431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:23.628139 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.628614 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.628615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.628615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.628668 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.628984 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.628985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.628985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.629093 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:23.629181 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.629503 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.629609 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:13:23.629956 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.620741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40421 (local address 127.0.98.1:55181)
0924 00:13:23.620901 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.620944 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.621849 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.622088 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.622091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.622344 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:23.622534 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.622538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.623264 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.623265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.624082 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.624083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.624426 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.624427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.624520 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.624997 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:23.625005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:23.626338 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:23.628219 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.628225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.628227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.628530 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.628736 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.628737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.628737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.628885 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.629173 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:23.629180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:23.629356 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.629475 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.629656 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260924 00:13:23.630641 18645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40421
I20260924 00:13:23.630759 18645 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:23.630964 18645 heartbeater.cc:511] Master 127.0.98.62:40421 requested a full tablet report, sending...
I20260924 00:13:23.631553 18454 ts_manager.cc:194] Registered new tserver with Master: 4376db40709e4c4fb20b7a1806a29bfd (127.0.98.1:39151)
I20260924 00:13:23.632175 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55181
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:13:23.653972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:23.735653 18650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:23.735888 18650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:23.735925 18650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:23.737134 18650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:23.737207 18650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:23.737253 18650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:23.738758 18650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:23.739020 18650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:23.739274 18650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:23.744421 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:23.744417 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.744550 18650 server_base.cc:1034] running on GCE node
W20260924 00:13:23.744421 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.744921 18650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:23.745240 18650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:23.746414 18650 hybrid_clock.cc:648] HybridClock initialized: now 1790208803746399 us; error 45 us; skew 500 ppm
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:23.749132 18650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:13:23.749631 18650 webserver.cc:460] Webserver started at http://127.0.98.2:39941/ using document root <none> and password file <none>
I20260924 00:13:23.749833 18650 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:23.749876 18650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:23.749971 18650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:23.751112 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "88f70d8627d547b2abce86b0d1eed6c1"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "cd105437c28326a5d190f5d690592eec"
server_key_iv: "211c2926947ded295716fd882813bce0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.751514 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "88f70d8627d547b2abce86b0d1eed6c1"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "cd105437c28326a5d190f5d690592eec"
server_key_iv: "211c2926947ded295716fd882813bce0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.752874 18650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:23.753651 18665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.753878 18650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:23.753968 18650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "88f70d8627d547b2abce86b0d1eed6c1"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "cd105437c28326a5d190f5d690592eec"
server_key_iv: "211c2926947ded295716fd882813bce0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.754015 18650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:23.775507 18650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:23.777438 18650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:23.777637 18650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:23.777868 18650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:23.778182 18650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:23.778236 18650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.778272 18650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:23.778301 18650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.783903 18650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39563
I20260924 00:13:23.783977 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39563 every 8 connection(s)
I20260924 00:13:23.784305 18650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:23.788811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18650
I20260924 00:13:23.788908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:23.789139 392 external_mini_cluster.cc:1428] Setting key e73a7e1de8a90c8ffbbadffcba7304c6
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:23.794322 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.785051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43087 (local address 127.0.98.62:40421)
0924 00:13:23.785129 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.785132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.785288 (+ 156us) server_negotiation.cc:407] Connection header received
0924 00:13:23.786208 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.786210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.786220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.786270 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:23.786844 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.787408 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.788426 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.788665 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.788738 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.791370 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:23.791372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:23.791372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:23.791380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:23.792789 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.793308 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.793310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.793310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.793373 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.793645 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.793646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.793646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.793764 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:23.793838 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.794126 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.794212 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:13:23.794502 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.784996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40421 (local address 127.0.98.2:43087)
0924 00:13:23.785163 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.785199 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.786110 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.786324 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.786327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.786579 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:23.786772 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.786776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.787516 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.787517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.788343 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.788345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.788754 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.788755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.788902 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.789488 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:23.789514 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:23.791270 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:23.792910 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.792917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.792919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.793226 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.793440 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.793441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.793441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.793559 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.793829 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:23.793835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:23.793979 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.794104 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.794265 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260924 00:13:23.795197 18779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40421
I20260924 00:13:23.795338 18779 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:23.795516 18779 heartbeater.cc:511] Master 127.0.98.62:40421 requested a full tablet report, sending...
I20260924 00:13:23.795948 18454 ts_manager.cc:194] Registered new tserver with Master: 88f70d8627d547b2abce86b0d1eed6c1 (127.0.98.2:39563)
I20260924 00:13:23.796468 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43087
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:13:23.816679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:23.898319 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:23.898519 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:23.898545 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:23.899794 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:23.899838 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:23.899865 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:23.901340 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:23.901574 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:23.901798 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:23.906834 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:23.906842 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.906967 18784 server_base.cc:1034] running on GCE node
W20260924 00:13:23.906842 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:23.907330 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:23.907578 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:23.908736 18784 hybrid_clock.cc:648] HybridClock initialized: now 1790208803908725 us; error 29 us; skew 500 ppm
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:23.911405 18784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:13:23.911859 18784 webserver.cc:460] Webserver started at http://127.0.98.3:45183/ using document root <none> and password file <none>
I20260924 00:13:23.912048 18784 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:23.912089 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:23.912173 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:23.913270 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "83f989af4b82456a8c53dba0ff82a828"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f3df49997fac3368696ccb5b1c0a572b"
server_key_iv: "4c4147f5ba61963220bb1f5fc5e8c2b0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.913568 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83f989af4b82456a8c53dba0ff82a828"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f3df49997fac3368696ccb5b1c0a572b"
server_key_iv: "4c4147f5ba61963220bb1f5fc5e8c2b0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.914832 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:13:23.915585 18799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.915746 18784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.915834 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "83f989af4b82456a8c53dba0ff82a828"
format_stamp: "Formatted at 2026-09-24 00:13:23 on dist-test-slave-3q8k"
server_key: "f3df49997fac3368696ccb5b1c0a572b"
server_key_iv: "4c4147f5ba61963220bb1f5fc5e8c2b0"
server_key_version: "encryptionkey@0"
I20260924 00:13:23.915898 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:23.928427 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:23.930946 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:23.931058 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:23.931283 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:23.931576 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:23.931605 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.931627 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:23.931639 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:23.937410 18784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36495
I20260924 00:13:23.937542 18912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36495 every 8 connection(s)
I20260924 00:13:23.937774 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:13:23.941740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18784
I20260924 00:13:23.941836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:23.942011 392 external_mini_cluster.cc:1428] Setting key d9f563b3558619424346e17136207d01
Sep 24 00:13:23 dist-test-slave-3q8k krb5kdc[18409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:23.947069 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.938535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43087 (local address 127.0.98.62:40421)
0924 00:13:23.938607 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.938609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.938783 (+ 174us) server_negotiation.cc:407] Connection header received
0924 00:13:23.939822 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.939823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.939833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.939898 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:23.940499 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.940993 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.942269 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.942502 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.942596 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.944736 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:23.944737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:23.944737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:23.944743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:23.945908 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.946285 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.946286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.946286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.946339 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.946554 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.946555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.946555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.946666 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:23.946716 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.946944 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.947005 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:13:23.947228 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.938464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40421 (local address 127.0.98.3:43087)
0924 00:13:23.938643 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.938697 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.939721 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.939972 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.939976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.940232 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:23.940421 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.940424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.941076 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.941076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.942166 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.942169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.942588 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.942589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.942738 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.943294 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:23.943305 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:23.944648 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:23.945978 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.945984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.945985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.946215 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.946401 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:23.946402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:23.946402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:23.946498 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:23.946722 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:23.946726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:23.946836 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.946917 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.947033 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260924 00:13:23.947721 18913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40421
I20260924 00:13:23.947813 18913 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:23.947950 18913 heartbeater.cc:511] Master 127.0.98.62:40421 requested a full tablet report, sending...
I20260924 00:13:23.948288 18454 ts_manager.cc:194] Registered new tserver with Master: 83f989af4b82456a8c53dba0ff82a828 (127.0.98.3:36495)
I20260924 00:13:23.948676 18454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43087
I20260924 00:13:23.953907 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:23.962028 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.955323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.98.62:40421)
0924 00:13:23.955389 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.955391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.955535 (+ 144us) server_negotiation.cc:407] Connection header received
0924 00:13:23.955618 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.955619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.955629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.955685 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:23.956296 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.956864 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.957624 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.957861 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.957947 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.959287 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:23.959288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:23.959288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:23.959296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:23.960866 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.961319 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.961320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.961320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.961373 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:23.961512 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:23.961512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:23.961513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:23.961635 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:23.961712 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.961903 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.961961 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:13:23.963704 18454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:23.965363 18454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:23.971912 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.967762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36495 (local address 127.0.0.1:58610)
0924 00:13:23.968092 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.968104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.968158 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.968471 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.968474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.968476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:23.968549 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.968550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.969518 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.969519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.970661 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.970663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.971569 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.971648 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.971671 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.971757 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.971810 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260924 00:13:23.972064 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.967720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39151 (local address 127.0.0.1:50118)
0924 00:13:23.967936 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.967991 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.968063 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.968543 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.968544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.968547 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:23.968626 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.968628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.970003 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.970004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.970966 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.970968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.971776 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.971851 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.971855 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.971918 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.971975 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260924 00:13:23.972334 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.967924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.98.3:36495)
0924 00:13:23.968274 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.968279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.968290 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:23.968335 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.968337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.968354 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.968459 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:23.968628 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.969416 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.970756 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.971485 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971557 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.971623 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.972191 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.972235 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":80,"threads_started":1}
I20260924 00:13:23.972409 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.967992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39563 (local address 127.0.0.1:39146)
0924 00:13:23.968309 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:23.968319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:23.968362 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:23.968752 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:23.968753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:23.968756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:23.968796 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.968797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.970020 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.970021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.971103 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:23.971105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.972170 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.972171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:23.972242 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.972246 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:23.972298 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:13:23.972335 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":61,"threads_started":1}
I20260924 00:13:23.972656 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.968124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.98.2:39563)
0924 00:13:23.968301 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.968586 (+ 285us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.968602 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:13:23.968645 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.968647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.968665 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.968744 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:23.968858 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.969903 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971197 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.971980 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.972042 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.972132 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.972500 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.972545 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260924 00:13:23.972662 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:23.968137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.0.98.1:39151)
0924 00:13:23.968363 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:23.968368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:23.968379 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:23.968420 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:23.968422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:23.968440 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:23.968709 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:23.968796 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.969885 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971045 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:23.971690 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:23.971765 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:23.971822 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:23.971947 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 00:13:23.972330 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260924 00:13:23.973110 18847 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea (DEFAULT_TABLE table=test-table [id=156963291b4c493a8f74298b4b8affe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:23.973197 18713 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea (DEFAULT_TABLE table=test-table [id=156963291b4c493a8f74298b4b8affe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:23.973299 18579 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea (DEFAULT_TABLE table=test-table [id=156963291b4c493a8f74298b4b8affe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:23.973492 18713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca7237274742f8aef7d7f5a3bf32ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:23.973500 18847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca7237274742f8aef7d7f5a3bf32ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:23.973691 18579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca7237274742f8aef7d7f5a3bf32ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:23.976496 18930 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Bootstrap starting.
I20260924 00:13:23.976958 18931 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Bootstrap starting.
I20260924 00:13:23.977257 18930 tablet_bootstrap.cc:654] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:23.977617 18931 tablet_bootstrap.cc:654] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:23.977841 18930 log.cc:824] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:23.978221 18931 log.cc:824] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:23.978240 18932 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Bootstrap starting.
I20260924 00:13:23.978659 18930 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: No bootstrap required, opened a new log
I20260924 00:13:23.978734 18930 ts_tablet_manager.cc:1397] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:23.978842 18932 tablet_bootstrap.cc:654] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:23.978993 18931 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: No bootstrap required, opened a new log
I20260924 00:13:23.979056 18931 ts_tablet_manager.cc:1397] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:23.979702 18932 log.cc:824] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:23.980472 18932 tablet_bootstrap.cc:492] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: No bootstrap required, opened a new log
I20260924 00:13:23.980545 18932 ts_tablet_manager.cc:1397] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:23.980561 18930 raft_consensus.cc:348] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.980763 18930 raft_consensus.cc:374] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:23.980805 18930 raft_consensus.cc:729] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4376db40709e4c4fb20b7a1806a29bfd, State: Initialized, Role: FOLLOWER
I20260924 00:13:23.980921 18930 consensus_queue.cc:260] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.981206 18930 ts_tablet_manager.cc:1428] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:13:23.981135 18931 raft_consensus.cc:348] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.981309 18931 raft_consensus.cc:374] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:23.981338 18645 heartbeater.cc:503] Master 127.0.98.62:40421 was elected leader, sending a full tablet report...
I20260924 00:13:23.981349 18931 raft_consensus.cc:729] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f70d8627d547b2abce86b0d1eed6c1, State: Initialized, Role: FOLLOWER
I20260924 00:13:23.981455 18931 consensus_queue.cc:260] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.981709 18931 ts_tablet_manager.cc:1428] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:23.981850 18779 heartbeater.cc:503] Master 127.0.98.62:40421 was elected leader, sending a full tablet report...
I20260924 00:13:23.982620 18932 raft_consensus.cc:348] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.982801 18932 raft_consensus.cc:374] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:23.982846 18932 raft_consensus.cc:729] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f989af4b82456a8c53dba0ff82a828, State: Initialized, Role: FOLLOWER
I20260924 00:13:23.982952 18932 consensus_queue.cc:260] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:23.983282 18932 ts_tablet_manager.cc:1428] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:23.983278 18913 heartbeater.cc:503] Master 127.0.98.62:40421 was elected leader, sending a full tablet report...
W20260924 00:13:24.034862 18780 tablet.cc:2369] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:24.080612 18938 raft_consensus.cc:482] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:24.080740 18938 raft_consensus.cc:504] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:24.081143 18938 leader_election.cc:288] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4376db40709e4c4fb20b7a1806a29bfd (127.0.98.1:39151), 88f70d8627d547b2abce86b0d1eed6c1 (127.0.98.2:39563)
I20260924 00:13:24.085093 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.081322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60235 (local address 127.0.98.2:39563)
0924 00:13:24.081397 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.081399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.081410 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:24.081444 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.081445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.081459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.081546 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.081723 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.082481 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084165 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.084725 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084791 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.084801 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.084982 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.085022 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:13:24.085098 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.081409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45025 (local address 127.0.98.1:39151)
0924 00:13:24.081451 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.081453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.081583 (+ 130us) server_negotiation.cc:407] Connection header received
0924 00:13:24.081618 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.081618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.081628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.081665 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.081829 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.082505 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084139 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.084710 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084790 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.084801 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.084982 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.085022 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:13:24.085099 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.081266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39151 (local address 127.0.98.3:45025)
0924 00:13:24.081441 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.081450 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.081546 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.081693 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.081694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.081696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.081758 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.081759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.082596 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.082597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.083991 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.083994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.084810 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.084891 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.084895 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.084958 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.085000 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":78,"thread_start_us":104,"threads_started":1}
I20260924 00:13:24.085393 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.081252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39563 (local address 127.0.98.3:60235)
0924 00:13:24.081317 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.081331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.081382 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.081546 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.081548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.081549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.081631 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.081633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.082596 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.082597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.084082 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.084084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.084810 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.084811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.084898 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.084900 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.084966 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.085077 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":162}
I20260924 00:13:24.085491 18599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca7237274742f8aef7d7f5a3bf32ea" candidate_uuid: "83f989af4b82456a8c53dba0ff82a828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4376db40709e4c4fb20b7a1806a29bfd" is_pre_election: true
I20260924 00:13:24.085695 18599 raft_consensus.cc:2454] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f989af4b82456a8c53dba0ff82a828 in term 0.
I20260924 00:13:24.085713 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca7237274742f8aef7d7f5a3bf32ea" candidate_uuid: "83f989af4b82456a8c53dba0ff82a828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f70d8627d547b2abce86b0d1eed6c1" is_pre_election: true
I20260924 00:13:24.085850 18733 raft_consensus.cc:2454] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f989af4b82456a8c53dba0ff82a828 in term 0.
I20260924 00:13:24.085937 18802 leader_election.cc:302] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4376db40709e4c4fb20b7a1806a29bfd, 83f989af4b82456a8c53dba0ff82a828; no voters:
I20260924 00:13:24.086086 18938 raft_consensus.cc:2781] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:24.086174 18938 raft_consensus.cc:482] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:24.086207 18938 raft_consensus.cc:3037] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.086895 18938 raft_consensus.cc:504] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:24.087059 18938 leader_election.cc:288] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [CANDIDATE]: Term 1 election: Requested vote from peers 4376db40709e4c4fb20b7a1806a29bfd (127.0.98.1:39151), 88f70d8627d547b2abce86b0d1eed6c1 (127.0.98.2:39563)
I20260924 00:13:24.087206 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca7237274742f8aef7d7f5a3bf32ea" candidate_uuid: "83f989af4b82456a8c53dba0ff82a828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f70d8627d547b2abce86b0d1eed6c1"
I20260924 00:13:24.087285 18599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca7237274742f8aef7d7f5a3bf32ea" candidate_uuid: "83f989af4b82456a8c53dba0ff82a828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4376db40709e4c4fb20b7a1806a29bfd"
I20260924 00:13:24.087319 18733 raft_consensus.cc:3037] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.087376 18599 raft_consensus.cc:3037] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.088074 18599 raft_consensus.cc:2454] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f989af4b82456a8c53dba0ff82a828 in term 1.
I20260924 00:13:24.088126 18733 raft_consensus.cc:2454] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f989af4b82456a8c53dba0ff82a828 in term 1.
I20260924 00:13:24.088280 18802 leader_election.cc:302] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4376db40709e4c4fb20b7a1806a29bfd, 83f989af4b82456a8c53dba0ff82a828; no voters:
I20260924 00:13:24.088379 18938 raft_consensus.cc:2781] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:24.088519 18938 raft_consensus.cc:686] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 1 LEADER]: Becoming Leader. State: Replica: 83f989af4b82456a8c53dba0ff82a828, State: Running, Role: LEADER
I20260924 00:13:24.088752 18938 consensus_queue.cc:237] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } }
I20260924 00:13:24.089733 18454 catalog_manager.cc:5495] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f989af4b82456a8c53dba0ff82a828 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "83f989af4b82456a8c53dba0ff82a828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f989af4b82456a8c53dba0ff82a828" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 } health_report { overall_health: UNKNOWN } } }
I20260924 00:13:24.116622 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.114090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.3:36495)
0924 00:13:24.114155 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.114158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.114178 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:13:24.114218 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.114219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.114246 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.114323 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:24.114466 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.114948 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.115790 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.115995 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.116082 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.116249 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:24.116348 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:24.116468 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.116499 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.116507 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:13:24.118517 18733 raft_consensus.cc:1261] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 1 FOLLOWER]: Refusing update from remote peer 83f989af4b82456a8c53dba0ff82a828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:24.118535 18599 raft_consensus.cc:1261] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd [term 1 FOLLOWER]: Refusing update from remote peer 83f989af4b82456a8c53dba0ff82a828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:24.118954 18940 consensus_queue.cc:1035] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f70d8627d547b2abce86b0d1eed6c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:24.119091 18938 consensus_queue.cc:1035] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4376db40709e4c4fb20b7a1806a29bfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:13:24.120731 18646 tablet.cc:2369] T a9ca7237274742f8aef7d7f5a3bf32ea P 4376db40709e4c4fb20b7a1806a29bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:24.121747 18943 mvcc.cc:205] Tried to move back new op lower bound from 7332695261666607104 to 7332695261548412928. Current Snapshot: MvccSnapshot[applied={T|T < 7332695261666607104}]
I20260924 00:13:24.124841 18454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:24.124949 18454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:24.125733 18454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de6860fe97764bf5afbabee0802caf86: Sending DeleteTablet for 3 replicas of tablet a9ca7237274742f8aef7d7f5a3bf32ea
I20260924 00:13:24.126113 18579 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50118
I20260924 00:13:24.126207 18847 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58610
I20260924 00:13:24.126273 18713 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ca7237274742f8aef7d7f5a3bf32ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39146
I20260924 00:13:24.126461 18954 tablet_replica.cc:330] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: stopping tablet replica
I20260924 00:13:24.126557 18954 raft_consensus.cc:2229] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:24.126670 18955 tablet_replica.cc:330] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: stopping tablet replica
I20260924 00:13:24.126698 18954 raft_consensus.cc:2258] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:24.126847 18955 raft_consensus.cc:2229] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:24.126931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18516
I20260924 00:13:24.127050 18955 raft_consensus.cc:2258] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:24.127148 18954 ts_tablet_manager.cc:1905] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:24.128458 18955 ts_tablet_manager.cc:1905] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:24.128945 18954 ts_tablet_manager.cc:1918] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:24.129021 18954 log.cc:1196] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/a9ca7237274742f8aef7d7f5a3bf32ea
I20260924 00:13:24.129371 18954 ts_tablet_manager.cc:1939] T a9ca7237274742f8aef7d7f5a3bf32ea P 83f989af4b82456a8c53dba0ff82a828: Deleting consensus metadata
I20260924 00:13:24.129846 18439 catalog_manager.cc:4841] TS 83f989af4b82456a8c53dba0ff82a828 (127.0.98.3:36495): tablet a9ca7237274742f8aef7d7f5a3bf32ea (table test-table [id=156963291b4c493a8f74298b4b8affe4]) successfully deleted
I20260924 00:13:24.130842 18955 ts_tablet_manager.cc:1918] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:24.130977 18955 log.cc:1196] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/a9ca7237274742f8aef7d7f5a3bf32ea
I20260924 00:13:24.131354 18955 ts_tablet_manager.cc:1939] T a9ca7237274742f8aef7d7f5a3bf32ea P 88f70d8627d547b2abce86b0d1eed6c1: Deleting consensus metadata
I20260924 00:13:24.131814 18439 catalog_manager.cc:4841] TS 88f70d8627d547b2abce86b0d1eed6c1 (127.0.98.2:39563): tablet a9ca7237274742f8aef7d7f5a3bf32ea (table test-table [id=156963291b4c493a8f74298b4b8affe4]) successfully deleted
W20260924 00:13:24.134514 18440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39151 (error 108)
I20260924 00:13:24.134747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18650
I20260924 00:13:24.135689 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.135539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39151 (local address 127.0.0.1:50130)
0924 00:13:24.135603 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.135646 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 00:13:24.135860 18440 catalog_manager.cc:4584] TS 4376db40709e4c4fb20b7a1806a29bfd (127.0.98.1:39151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9ca7237274742f8aef7d7f5a3bf32ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39151: connect: Connection refused (error 111)
I20260924 00:13:24.140390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18784
I20260924 00:13:24.145035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18422
2026-09-24T00:13:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2017 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T00:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:24Z Disabled control of system clock
I20260924 00:13:24.172354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:24.253008 18960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:24.253240 18960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:24.253261 18960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:24.254343 18960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:24.254376 18960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:24.254390 18960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:24.254401 18960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:24.254413 18960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:24.255708 18960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:24.255918 18960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:24.256239 18960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:24.261061 18966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.261135 18968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:24.261274 18960 server_base.cc:1034] running on GCE node
W20260924 00:13:24.261061 18965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:24.261561 18960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:24.261868 18960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:24.263034 18960 hybrid_clock.cc:648] HybridClock initialized: now 1790208804263019 us; error 57 us; skew 500 ppm
I20260924 00:13:24.264400 18960 webserver.cc:460] Webserver started at http://127.0.98.62:43715/ using document root <none> and password file <none>
I20260924 00:13:24.264671 18960 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:24.264746 18960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:24.264894 18960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:24.265964 18960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "f111cb763bea4ee1a87bffacdfdf5525"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "37bf7f4c8a188505929524cc46074666"
server_key_iv: "13e10b6b25eeca1b6b018041f109238e"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.266372 18960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "f111cb763bea4ee1a87bffacdfdf5525"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "37bf7f4c8a188505929524cc46074666"
server_key_iv: "13e10b6b25eeca1b6b018041f109238e"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.267732 18960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:24.268460 18974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.268647 18960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:24.268718 18960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "f111cb763bea4ee1a87bffacdfdf5525"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "37bf7f4c8a188505929524cc46074666"
server_key_iv: "13e10b6b25eeca1b6b018041f109238e"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.268769 18960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:24.277146 18960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:24.277410 18960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:24.277513 18960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:24.281620 18960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41957
I20260924 00:13:24.281652 19026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41957 every 8 connection(s)
I20260924 00:13:24.281977 18960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:24.282877 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:24.285239 19027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: Bootstrap starting.
I20260924 00:13:24.285822 19027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:24.286125 19027 log.cc:824] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:24.286568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 18960
I20260924 00:13:24.286742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:24.286923 392 external_mini_cluster.cc:1428] Setting key 1d955566a032af2fb8bf0ee66c2d6c4c
I20260924 00:13:24.286923 19027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: No bootstrap required, opened a new log
I20260924 00:13:24.289878 19027 raft_consensus.cc:348] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 00:13:24.290047 19027 raft_consensus.cc:374] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:24.290088 19027 raft_consensus.cc:729] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f111cb763bea4ee1a87bffacdfdf5525, State: Initialized, Role: FOLLOWER
I20260924 00:13:24.290199 19027 consensus_queue.cc:260] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 00:13:24.290263 19027 raft_consensus.cc:388] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:24.290293 19027 raft_consensus.cc:482] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:24.290339 19027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.290884 19027 raft_consensus.cc:504] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 00:13:24.290970 19027 leader_election.cc:302] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f111cb763bea4ee1a87bffacdfdf5525; no voters:
I20260924 00:13:24.291164 19027 leader_election.cc:288] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:24.291252 19032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:24.291394 19032 raft_consensus.cc:686] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [term 1 LEADER]: Becoming Leader. State: Replica: f111cb763bea4ee1a87bffacdfdf5525, State: Running, Role: LEADER
I20260924 00:13:24.291438 19027 sys_catalog.cc:564] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:24.291477 19032 consensus_queue.cc:237] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 00:13:24.291810 19032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f111cb763bea4ee1a87bffacdfdf5525. Latest consensus state: current_term: 1 leader_uuid: "f111cb763bea4ee1a87bffacdfdf5525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 00:13:24.291899 19032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:24.291819 19033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f111cb763bea4ee1a87bffacdfdf5525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f111cb763bea4ee1a87bffacdfdf5525" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 00:13:24.292009 19033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:24.292120 19039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:24.292587 19039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:24.293489 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.287527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.0.98.62:41957)
0924 00:13:24.287766 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.287776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.287814 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:13:24.288440 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.288449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.288526 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.288689 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:24.288864 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.290302 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.291146 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.291787 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.292324 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.292717 (+ 393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:24.292720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:24.292721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:24.292954 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:24.292999 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:24.293017 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:24.293075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.293188 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.293271 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260924 00:13:24.294606 19039 catalog_manager.cc:1347] Generated new cluster ID: 5498da5eb91343efba498a916188ea46
I20260924 00:13:24.294657 19039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:24.303156 19039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:24.303691 19039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:24.318693 19039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: Generated new TSK 0
I20260924 00:13:24.318918 19039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:24.328783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:24.420547 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:24.420768 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:24.420792 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:24.422060 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:24.422106 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:24.422161 19051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:24.423815 19051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:24.424085 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:24.424417 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:24.429847 19056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.429840 19059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.429889 19057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:24.430153 19051 server_base.cc:1034] running on GCE node
I20260924 00:13:24.430630 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:24.430836 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:24.431977 19051 hybrid_clock.cc:648] HybridClock initialized: now 1790208804431968 us; error 34 us; skew 500 ppm
I20260924 00:13:24.439002 19051 webserver.cc:460] Webserver started at http://127.0.98.1:36395/ using document root <none> and password file <none>
I20260924 00:13:24.439358 19051 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:24.439435 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:24.439620 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:24.440894 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc9b348cc6ce4e49bcab1631e91efee8"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "abaf061bcd15a886a25ecf0bd91b96f8"
server_key_iv: "ae986677cae1a57d9c10bb172c871053"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.441262 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9b348cc6ce4e49bcab1631e91efee8"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "abaf061bcd15a886a25ecf0bd91b96f8"
server_key_iv: "ae986677cae1a57d9c10bb172c871053"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.442596 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:24.443439 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.443635 19051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:24.443734 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "fc9b348cc6ce4e49bcab1631e91efee8"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "abaf061bcd15a886a25ecf0bd91b96f8"
server_key_iv: "ae986677cae1a57d9c10bb172c871053"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.443816 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:24.456301 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:24.456632 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:24.456792 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:24.457029 19051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:24.457386 19051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:24.457442 19051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.457485 19051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:24.457509 19051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.463295 19051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39303
I20260924 00:13:24.463351 19178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39303 every 8 connection(s)
I20260924 00:13:24.463723 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:13:24.464071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19051
I20260924 00:13:24.464179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:24.464385 392 external_mini_cluster.cc:1428] Setting key 81852c31e73f82ac8874e521f331bcd2
I20260924 00:13:24.465468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:13:24.470091 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.464588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47089 (local address 127.0.98.62:41957)
0924 00:13:24.464636 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.464639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.464867 (+ 228us) server_negotiation.cc:407] Connection header received
0924 00:13:24.465403 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.465404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.465422 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.465499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:24.466147 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.466930 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.468533 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.468725 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.468778 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.469782 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:24.469784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:24.469784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:24.469788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:24.469802 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:24.469806 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:24.469858 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.470006 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.470027 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:24.470427 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.464558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.1:47089)
0924 00:13:24.464749 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.464790 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.465300 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.465553 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.465556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.465565 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:24.466057 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.466061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.467026 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.467027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.468415 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.468418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.468812 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.468813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.468979 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.469608 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:24.469609 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:24.469653 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:24.469688 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:24.469901 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:24.469909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:24.469913 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.470011 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.470143 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260924 00:13:24.470983 19179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 00:13:24.471123 19179 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:24.471387 19179 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 00:13:24.471982 18991 ts_manager.cc:194] Registered new tserver with Master: fc9b348cc6ce4e49bcab1631e91efee8 (127.0.98.1:39303)
I20260924 00:13:24.472653 18991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47089
W20260924 00:13:24.550006 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:24.550273 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:24.550311 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:24.551558 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:24.551635 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:24.551679 19182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:24.553223 19182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:24.553519 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:24.553833 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:24.559478 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.559478 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.559478 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:24.559799 19182 server_base.cc:1034] running on GCE node
I20260924 00:13:24.560065 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:24.560379 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:24.561589 19182 hybrid_clock.cc:648] HybridClock initialized: now 1790208804561572 us; error 55 us; skew 500 ppm
I20260924 00:13:24.563160 19182 webserver.cc:460] Webserver started at http://127.0.98.2:38069/ using document root <none> and password file <none>
I20260924 00:13:24.563482 19182 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:24.563567 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:24.563736 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:24.564915 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "a5673276e1c54507bed42cb00bccf856"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "fb79e4a5046729f9a165e0baea0be477"
server_key_iv: "346ae85e008b0ef5d5fece970fae14d4"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.565301 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a5673276e1c54507bed42cb00bccf856"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "fb79e4a5046729f9a165e0baea0be477"
server_key_iv: "346ae85e008b0ef5d5fece970fae14d4"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.566738 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:24.567790 19196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.568051 19182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:13:24.568166 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "a5673276e1c54507bed42cb00bccf856"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "fb79e4a5046729f9a165e0baea0be477"
server_key_iv: "346ae85e008b0ef5d5fece970fae14d4"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.568249 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:24.599520 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:24.599860 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:24.600023 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:24.600265 19182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:24.600608 19182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:24.600662 19182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.600706 19182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:24.600734 19182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.606669 19182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37151
I20260924 00:13:24.606719 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37151 every 8 connection(s)
I20260924 00:13:24.607069 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:24.611404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19182
I20260924 00:13:24.611514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:24.611737 392 external_mini_cluster.cc:1428] Setting key d153ce8f2e4d03d38b4fca90c021ce5d
I20260924 00:13:24.612771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:13:24.612802 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.607861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58977 (local address 127.0.98.62:41957)
0924 00:13:24.607932 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.607934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.608238 (+ 304us) server_negotiation.cc:407] Connection header received
0924 00:13:24.608799 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.608800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.608813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.608870 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:24.609572 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.610074 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.611072 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.611364 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.611441 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.612524 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:24.612525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:24.612526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:24.612530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:24.612546 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:24.612550 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:24.612601 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.612696 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.612723 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:13:24.613019 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.607857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.2:58977)
0924 00:13:24.608113 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.608148 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.608679 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.608941 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.608945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.608956 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:24.609467 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.609471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.610167 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.610168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.610976 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.610977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.611527 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.611528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.611692 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.612357 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:24.612357 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:24.612398 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:24.612429 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:24.612611 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:24.612616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:24.612619 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.612690 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.612796 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":141,"threads_started":1}
I20260924 00:13:24.613543 19310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 00:13:24.613639 19310 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:24.613818 19310 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 00:13:24.614210 18991 ts_manager.cc:194] Registered new tserver with Master: a5673276e1c54507bed42cb00bccf856 (127.0.98.2:37151)
I20260924 00:13:24.614595 18991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58977
W20260924 00:13:24.694487 19313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:24.694703 19313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:24.694725 19313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:24.695915 19313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:24.695960 19313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:24.695986 19313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:24.697403 19313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:24.697669 19313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:24.697980 19313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:24.703155 19321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.703162 19319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:24.703183 19318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:24.703513 19313 server_base.cc:1034] running on GCE node
I20260924 00:13:24.703698 19313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:24.703919 19313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:24.705165 19313 hybrid_clock.cc:648] HybridClock initialized: now 1790208804705164 us; error 51 us; skew 500 ppm
I20260924 00:13:24.706444 19313 webserver.cc:460] Webserver started at http://127.0.98.3:36207/ using document root <none> and password file <none>
I20260924 00:13:24.706744 19313 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:24.706825 19313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:24.706975 19313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:24.708179 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b440b05082e4b269c76519ae57289ea"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "8024511b1aaeb6edb437bdc5ac216770"
server_key_iv: "176e4358c4d3ebf9a936646b7143abdb"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.708549 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b440b05082e4b269c76519ae57289ea"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "8024511b1aaeb6edb437bdc5ac216770"
server_key_iv: "176e4358c4d3ebf9a936646b7143abdb"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.710052 19313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:13:24.710911 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.711156 19313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:24.711314 19313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "5b440b05082e4b269c76519ae57289ea"
format_stamp: "Formatted at 2026-09-24 00:13:24 on dist-test-slave-3q8k"
server_key: "8024511b1aaeb6edb437bdc5ac216770"
server_key_iv: "176e4358c4d3ebf9a936646b7143abdb"
server_key_version: "encryptionkey@0"
I20260924 00:13:24.711400 19313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:24.736470 19313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:24.736828 19313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:24.737003 19313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:24.737251 19313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:24.737601 19313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:24.737660 19313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.737702 19313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:24.737727 19313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:24.743763 19313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38371
I20260924 00:13:24.743808 19440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38371 every 8 connection(s)
I20260924 00:13:24.744153 19313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:13:24.748230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19313
I20260924 00:13:24.748334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:24.748528 392 external_mini_cluster.cc:1428] Setting key aa0e7b3130849cc79e1d97ef860b4d5a
I20260924 00:13:24.750109 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.744948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44743 (local address 127.0.98.62:41957)
0924 00:13:24.745073 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.745075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.745279 (+ 204us) server_negotiation.cc:407] Connection header received
0924 00:13:24.745890 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.745892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.745905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.745954 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:24.746665 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.747184 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.748332 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.748575 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.748659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.749768 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:24.749769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:24.749769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:24.749803 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:24.749822 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:24.749826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:24.749894 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.749972 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.750051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:13:24.750276 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.744908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.3:44743)
0924 00:13:24.745152 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.745189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.745780 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.746035 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.746038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.746045 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:24.746565 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.746570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.747327 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.747328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.748207 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.748209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.748670 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.748671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.748856 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.749559 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:24.749559 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:24.749600 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:24.749628 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:24.749898 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:24.749903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:24.749905 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.749962 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.750052 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260924 00:13:24.750756 19441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 00:13:24.750836 19441 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:24.750976 19441 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 00:13:24.751380 18991 ts_manager.cc:194] Registered new tserver with Master: 5b440b05082e4b269c76519ae57289ea (127.0.98.3:38371)
I20260924 00:13:24.751716 18991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44743
I20260924 00:13:24.760924 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:24.765584 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.762431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.98.62:41957)
0924 00:13:24.762509 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.762511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.762608 (+ 97us) server_negotiation.cc:407] Connection header received
0924 00:13:24.762703 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.762704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.762720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.762782 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:24.762966 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.763493 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.764249 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.764438 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.764528 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.765252 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:24.765254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:24.765255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:24.765259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:24.765273 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:24.765276 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:24.765409 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.765475 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.765486 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:13:24.767427 18991 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:24.769096 18991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:24.775542 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38371 (local address 127.0.0.1:58566)
0924 00:13:24.771472 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.771483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.771533 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.772187 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.772188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.772190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.772238 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.772239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773155 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.773156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.774507 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.774509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775211 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775232 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.775309 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.775322 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.775398 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.775454 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":38,"thread_start_us":135,"threads_started":1}
I20260924 00:13:24.775555 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.0.98.3:38371)
0924 00:13:24.771910 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.771915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.771937 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:13:24.771997 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.771999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.772015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.772098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.772311 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773068 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.774619 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775105 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775206 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.775328 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.775416 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.775452 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":90,"threads_started":1}
I20260924 00:13:24.775580 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37151 (local address 127.0.0.1:35632)
0924 00:13:24.771526 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.771535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.771599 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.772105 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.772106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.772107 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.772139 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.772140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773430 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.773431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.774337 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.774339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775317 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.775385 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.775390 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.775419 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.775462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260924 00:13:24.775928 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.0.98.2:37151)
0924 00:13:24.771875 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.771880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.771889 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:24.771936 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.771938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.772005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.772203 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.772258 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773346 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.774434 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775170 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775688 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.775799 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.775822 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.775857 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260924 00:13:24.776119 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39303 (local address 127.0.0.1:43818)
0924 00:13:24.771344 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.771387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.771455 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.771856 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.771860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.771862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.771951 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.771953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773287 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.773288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.774507 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.774509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775387 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.775441 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.775444 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.775471 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.775501 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":12,"thread_start_us":79,"threads_started":1}
I20260924 00:13:24.776490 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.771479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.1:39303)
0924 00:13:24.771638 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.771644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.771655 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:24.771696 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.771697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.771714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.771835 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.772055 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.773208 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.774594 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.775306 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.775580 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.776344 (+ 764us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.776369 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.776416 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260924 00:13:24.776367 19375 tablet_service.cc:1467] Processing CreateTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 (DEFAULT_TABLE table=test-table [id=6e0e1748311241dc851d48efa6f907f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:24.776597 19244 tablet_service.cc:1467] Processing CreateTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 (DEFAULT_TABLE table=test-table [id=6e0e1748311241dc851d48efa6f907f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:24.776737 19375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd124338546f4a309b709cbb6d2cb4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:24.776857 19244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd124338546f4a309b709cbb6d2cb4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:24.779704 19455 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Bootstrap starting.
I20260924 00:13:24.779834 19113 tablet_service.cc:1467] Processing CreateTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 (DEFAULT_TABLE table=test-table [id=6e0e1748311241dc851d48efa6f907f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:24.780184 19113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd124338546f4a309b709cbb6d2cb4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:24.780392 19455 tablet_bootstrap.cc:654] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:24.780705 19455 log.cc:824] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:24.781507 19455 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: No bootstrap required, opened a new log
I20260924 00:13:24.781585 19455 ts_tablet_manager.cc:1397] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:13:24.783417 19456 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Bootstrap starting.
I20260924 00:13:24.783658 19458 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Bootstrap starting.
I20260924 00:13:24.784185 19456 tablet_bootstrap.cc:654] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:24.784125 19455 raft_consensus.cc:348] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.784297 19455 raft_consensus.cc:374] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:24.784345 19455 raft_consensus.cc:729] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5673276e1c54507bed42cb00bccf856, State: Initialized, Role: FOLLOWER
I20260924 00:13:24.784380 19458 tablet_bootstrap.cc:654] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:24.784466 19455 consensus_queue.cc:260] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.784592 19456 log.cc:824] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:24.784698 19458 log.cc:824] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:24.784762 19455 ts_tablet_manager.cc:1428] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:13:24.785130 19310 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 00:13:24.785456 19458 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: No bootstrap required, opened a new log
I20260924 00:13:24.785529 19458 ts_tablet_manager.cc:1397] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:13:24.785647 19456 tablet_bootstrap.cc:492] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: No bootstrap required, opened a new log
I20260924 00:13:24.785707 19456 ts_tablet_manager.cc:1397] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:24.788260 19456 raft_consensus.cc:348] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.788219 19458 raft_consensus.cc:348] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.788403 19456 raft_consensus.cc:374] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:24.788403 19458 raft_consensus.cc:374] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:24.788437 19458 raft_consensus.cc:729] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9b348cc6ce4e49bcab1631e91efee8, State: Initialized, Role: FOLLOWER
I20260924 00:13:24.788437 19456 raft_consensus.cc:729] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b440b05082e4b269c76519ae57289ea, State: Initialized, Role: FOLLOWER
I20260924 00:13:24.788554 19456 consensus_queue.cc:260] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.788569 19458 consensus_queue.cc:260] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.788827 19458 ts_tablet_manager.cc:1428] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:13:24.788827 19456 ts_tablet_manager.cc:1428] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:24.788987 19179 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 00:13:24.789044 19441 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
W20260924 00:13:24.857738 19311 tablet.cc:2369] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:13:24.964577 19180 tablet.cc:2369] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:24.968286 19463 raft_consensus.cc:482] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:24.968405 19463 raft_consensus.cc:504] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.968715 19463 leader_election.cc:288] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b440b05082e4b269c76519ae57289ea (127.0.98.3:38371), a5673276e1c54507bed42cb00bccf856 (127.0.98.2:37151)
I20260924 00:13:24.971943 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.969022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38371 (local address 127.0.98.1:33565)
0924 00:13:24.969102 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.969116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.969185 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.969359 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.969360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.969362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.969439 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.969440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.970071 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.970071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.971020 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.971023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.971675 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.971747 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.971750 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.971802 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.971834 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 00:13:24.971982 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.969188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33565 (local address 127.0.98.3:38371)
0924 00:13:24.969235 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.969237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.969244 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:24.969282 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.969282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.969289 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.969356 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.969510 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.969977 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971116 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.971537 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971656 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.971669 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.971823 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.971872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:13:24.972052 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.969215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47509 (local address 127.0.98.2:37151)
0924 00:13:24.969273 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:24.969275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:24.969369 (+ 94us) server_negotiation.cc:407] Connection header received
0924 00:13:24.969464 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:24.969465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:24.969481 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:24.969544 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:24.969703 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.970170 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971201 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.971636 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971710 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.971719 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:24.971904 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:13:24.971964 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:13:24.972064 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:24.969136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37151 (local address 127.0.98.1:47509)
0924 00:13:24.969277 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:24.969287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:24.969379 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:24.969548 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:24.969549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:24.969550 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:24.969630 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.969632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.970259 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.970260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.971112 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:24.971114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:24.971719 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:24.971720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:24.971823 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:24.971828 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:13:24.971884 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:13:24.971930 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260924 00:13:24.972560 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd124338546f4a309b709cbb6d2cb4f2" candidate_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b440b05082e4b269c76519ae57289ea" is_pre_election: true
I20260924 00:13:24.972559 19264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd124338546f4a309b709cbb6d2cb4f2" candidate_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5673276e1c54507bed42cb00bccf856" is_pre_election: true
I20260924 00:13:24.972729 19264 raft_consensus.cc:2454] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9b348cc6ce4e49bcab1631e91efee8 in term 0.
I20260924 00:13:24.972723 19395 raft_consensus.cc:2454] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9b348cc6ce4e49bcab1631e91efee8 in term 0.
I20260924 00:13:24.972972 19069 leader_election.cc:302] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b440b05082e4b269c76519ae57289ea, fc9b348cc6ce4e49bcab1631e91efee8; no voters:
I20260924 00:13:24.973142 19463 raft_consensus.cc:2781] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:24.973218 19463 raft_consensus.cc:482] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:24.973248 19463 raft_consensus.cc:3037] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.973980 19463 raft_consensus.cc:504] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.974087 19463 leader_election.cc:288] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [CANDIDATE]: Term 1 election: Requested vote from peers 5b440b05082e4b269c76519ae57289ea (127.0.98.3:38371), a5673276e1c54507bed42cb00bccf856 (127.0.98.2:37151)
I20260924 00:13:24.974308 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd124338546f4a309b709cbb6d2cb4f2" candidate_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b440b05082e4b269c76519ae57289ea"
I20260924 00:13:24.974383 19395 raft_consensus.cc:3037] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.974351 19264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd124338546f4a309b709cbb6d2cb4f2" candidate_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5673276e1c54507bed42cb00bccf856"
I20260924 00:13:24.974472 19264 raft_consensus.cc:3037] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:24.975263 19395 raft_consensus.cc:2454] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9b348cc6ce4e49bcab1631e91efee8 in term 1.
I20260924 00:13:24.975365 19264 raft_consensus.cc:2454] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9b348cc6ce4e49bcab1631e91efee8 in term 1.
I20260924 00:13:24.975431 19069 leader_election.cc:302] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b440b05082e4b269c76519ae57289ea, fc9b348cc6ce4e49bcab1631e91efee8; no voters:
I20260924 00:13:24.975541 19463 raft_consensus.cc:2781] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:24.975718 19463 raft_consensus.cc:686] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 1 LEADER]: Becoming Leader. State: Replica: fc9b348cc6ce4e49bcab1631e91efee8, State: Running, Role: LEADER
I20260924 00:13:24.975821 19463 consensus_queue.cc:237] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } }
I20260924 00:13:24.976724 18990 catalog_manager.cc:5495] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc9b348cc6ce4e49bcab1631e91efee8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9b348cc6ce4e49bcab1631e91efee8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 } health_report { overall_health: UNKNOWN } } }
W20260924 00:13:24.994735 19442 tablet.cc:2369] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:25.004570 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:25.001994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.98.1:39303)
0924 00:13:25.002056 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:25.002058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:25.002093 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:13:25.002169 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:25.002170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:25.002191 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:25.002256 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:25.002379 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:25.002863 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:25.003704 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:25.003893 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:25.003952 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:25.004197 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:25.004286 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:25.004363 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:25.004478 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 00:13:25.004488 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:13:25.006428 19264 raft_consensus.cc:1261] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 1 FOLLOWER]: Refusing update from remote peer fc9b348cc6ce4e49bcab1631e91efee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:25.006451 19395 raft_consensus.cc:1261] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 1 FOLLOWER]: Refusing update from remote peer fc9b348cc6ce4e49bcab1631e91efee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:25.006829 19463 consensus_queue.cc:1035] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b440b05082e4b269c76519ae57289ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:25.007058 19465 consensus_queue.cc:1035] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5673276e1c54507bed42cb00bccf856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:25.009850 19469 mvcc.cc:205] Tried to move back new op lower bound from 7332695265303650304 to 7332695265181753344. Current Snapshot: MvccSnapshot[applied={T|T < 7332695265303650304}]
I20260924 00:13:25.012118 18991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:25.012200 18991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:25.013003 18991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f111cb763bea4ee1a87bffacdfdf5525: Sending DeleteTablet for 3 replicas of tablet bd124338546f4a309b709cbb6d2cb4f2
I20260924 00:13:25.013402 19375 tablet_service.cc:1514] Processing DeleteTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:25 UTC) from {username='slave'} at 127.0.0.1:58566
I20260924 00:13:25.013427 19244 tablet_service.cc:1514] Processing DeleteTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:25 UTC) from {username='slave'} at 127.0.0.1:35632
I20260924 00:13:25.013620 19113 tablet_service.cc:1514] Processing DeleteTablet for tablet bd124338546f4a309b709cbb6d2cb4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:25 UTC) from {username='slave'} at 127.0.0.1:43818
I20260924 00:13:25.013660 19478 tablet_replica.cc:330] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: stopping tablet replica
I20260924 00:13:25.013762 19478 raft_consensus.cc:2229] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:25.013887 19480 tablet_replica.cc:330] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8: stopping tablet replica
I20260924 00:13:25.013871 19478 raft_consensus.cc:2258] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:25.014055 19480 raft_consensus.cc:2229] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:25.014160 19479 tablet_replica.cc:330] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: stopping tablet replica
I20260924 00:13:25.014256 19479 raft_consensus.cc:2229] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:25.014247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19051
I20260924 00:13:25.014269 19480 raft_consensus.cc:2258] T bd124338546f4a309b709cbb6d2cb4f2 P fc9b348cc6ce4e49bcab1631e91efee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:25.014360 19479 raft_consensus.cc:2258] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:25.015460 19479 ts_tablet_manager.cc:1905] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:25.017073 19478 ts_tablet_manager.cc:1905] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:25.017447 19479 ts_tablet_manager.cc:1918] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:25.017534 19479 log.cc:1196] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/bd124338546f4a309b709cbb6d2cb4f2
I20260924 00:13:25.017784 19479 ts_tablet_manager.cc:1939] T bd124338546f4a309b709cbb6d2cb4f2 P a5673276e1c54507bed42cb00bccf856: Deleting consensus metadata
I20260924 00:13:25.018249 18978 catalog_manager.cc:4841] TS a5673276e1c54507bed42cb00bccf856 (127.0.98.2:37151): tablet bd124338546f4a309b709cbb6d2cb4f2 (table test-table [id=6e0e1748311241dc851d48efa6f907f8]) successfully deleted
I20260924 00:13:25.018559 19478 ts_tablet_manager.cc:1918] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:25.018647 19478 log.cc:1196] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/bd124338546f4a309b709cbb6d2cb4f2
I20260924 00:13:25.018860 19478 ts_tablet_manager.cc:1939] T bd124338546f4a309b709cbb6d2cb4f2 P 5b440b05082e4b269c76519ae57289ea: Deleting consensus metadata
I20260924 00:13:25.019822 18978 catalog_manager.cc:4841] TS 5b440b05082e4b269c76519ae57289ea (127.0.98.3:38371): tablet bd124338546f4a309b709cbb6d2cb4f2 (table test-table [id=6e0e1748311241dc851d48efa6f907f8]) successfully deleted
W20260924 00:13:25.020398 18976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39303 (error 108)
I20260924 00:13:25.020880 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:25.020760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39303 (local address 127.0.0.1:43830)
0924 00:13:25.020814 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:25.020853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:13:25.020961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19182
W20260924 00:13:25.021034 18976 catalog_manager.cc:4584] TS fc9b348cc6ce4e49bcab1631e91efee8 (127.0.98.1:39303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd124338546f4a309b709cbb6d2cb4f2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39303: connect: Connection refused (error 111)
I20260924 00:13:25.026445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19313
I20260924 00:13:25.030954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 18960
2026-09-24T00:13:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (885 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:25 dist-test-slave-3q8k krb5kdc[19484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:25 dist-test-slave-3q8k krb5kdc[19484](info): set up 2 sockets
Sep 24 00:13:25 dist-test-slave-3q8k krb5kdc[19484](info): commencing operation
krb5kdc: starting...
W20260924 00:13:26.067175 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:26.186851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:26.267249 19497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:26.267503 19497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:26.267540 19497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:26.268721 19497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:26.268795 19497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:26.268819 19497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:26.268842 19497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:26.268870 19497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:26.270347 19497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:26.270645 19497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:26.270895 19497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:26.275791 19502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.275770 19505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:26.275803 19497 server_base.cc:1034] running on GCE node
W20260924 00:13:26.275770 19503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:26.276365 19497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:26.276657 19497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:26.277806 19497 hybrid_clock.cc:648] HybridClock initialized: now 1790208806277792 us; error 31 us; skew 500 ppm
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:26.280346 19497 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:26.280772 19497 webserver.cc:460] Webserver started at http://127.0.98.62:44697/ using document root <none> and password file <none>
I20260924 00:13:26.281016 19497 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:26.281080 19497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:26.281229 19497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:26.282285 19497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "7dab30be9b704d1ba6926608329d2d48"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "a36b5d766710fef24f96ecd092971def"
server_key_iv: "e541ab48f3db18ced14f34b51fbaf2a1"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.282649 19497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dab30be9b704d1ba6926608329d2d48"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "a36b5d766710fef24f96ecd092971def"
server_key_iv: "e541ab48f3db18ced14f34b51fbaf2a1"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.284036 19497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:26.284739 19512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.284902 19497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:26.285005 19497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7dab30be9b704d1ba6926608329d2d48"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "a36b5d766710fef24f96ecd092971def"
server_key_iv: "e541ab48f3db18ced14f34b51fbaf2a1"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.285079 19497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:26.312752 19497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:26.314558 19497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:26.314719 19497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:26.319157 19497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45917
I20260924 00:13:26.319195 19564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45917 every 8 connection(s)
I20260924 00:13:26.319589 19497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:26.320429 19565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:26.321627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19497
I20260924 00:13:26.321751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:26.321918 392 external_mini_cluster.cc:1428] Setting key 8941775c4d3ad4d865bcc6fab8bd37c5
I20260924 00:13:26.323057 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: Bootstrap starting.
I20260924 00:13:26.323864 19565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:26.324191 19565 log.cc:824] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:26.324898 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: No bootstrap required, opened a new log
I20260924 00:13:26.327138 19565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260924 00:13:26.327365 19565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:26.327415 19565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dab30be9b704d1ba6926608329d2d48, State: Initialized, Role: FOLLOWER
I20260924 00:13:26.327570 19565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260924 00:13:26.327665 19565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:26.327703 19565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:26.327756 19565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:26.328526 19565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260924 00:13:26.328645 19565 leader_election.cc:302] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dab30be9b704d1ba6926608329d2d48; no voters:
I20260924 00:13:26.328824 19565 leader_election.cc:288] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:26.328877 19570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:26.329043 19570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [term 1 LEADER]: Becoming Leader. State: Replica: 7dab30be9b704d1ba6926608329d2d48, State: Running, Role: LEADER
I20260924 00:13:26.329082 19565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:26.329134 19570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:26.329501 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dab30be9b704d1ba6926608329d2d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } } }
I20260924 00:13:26.329574 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:26.329777 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dab30be9b704d1ba6926608329d2d48. Latest consensus state: current_term: 1 leader_uuid: "7dab30be9b704d1ba6926608329d2d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dab30be9b704d1ba6926608329d2d48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } } }
I20260924 00:13:26.329980 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:26.330623 19576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:26.331255 19576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:26.332819 19576 catalog_manager.cc:1347] Generated new cluster ID: 6301656d798145adaf7e62c3c913f18f
I20260924 00:13:26.332868 19576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:26.334136 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.322436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.98.62:45917)
0924 00:13:26.322605 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.322613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.322638 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:13:26.323039 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.323046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.323089 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.323306 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:26.324134 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.325355 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.326417 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.326826 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.326919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.329953 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:26.329958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:26.329958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:26.329969 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:26.332071 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.332515 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.332518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.332518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.332628 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.332879 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.332880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.332880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.333185 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:26.333314 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.333780 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.333926 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260924 00:13:26.345139 19576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:26.345873 19576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:26.349659 19576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: Generated new TSK 0
I20260924 00:13:26.349836 19576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:13:26.376286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:26.458829 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:26.459043 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:26.459067 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:26.460248 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:26.460291 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:26.460317 19591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:26.461822 19591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:26.462046 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:26.462266 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:26.467262 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.467262 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.467275 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:26.467500 19591 server_base.cc:1034] running on GCE node
I20260924 00:13:26.467734 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:26.467993 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:26.469168 19591 hybrid_clock.cc:648] HybridClock initialized: now 1790208806469157 us; error 28 us; skew 500 ppm
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:26.471853 19591 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:13:26.472361 19591 webserver.cc:460] Webserver started at http://127.0.98.1:33711/ using document root <none> and password file <none>
I20260924 00:13:26.472546 19591 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:26.472586 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:26.472672 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:26.473686 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "96869395642d4b1fb27fe707692ff79f"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "af683006f86d2fad08b93683688a587e"
server_key_iv: "9666b284f749c864cc7cf826112b49c3"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.473997 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96869395642d4b1fb27fe707692ff79f"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "af683006f86d2fad08b93683688a587e"
server_key_iv: "9666b284f749c864cc7cf826112b49c3"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.475299 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:26.475992 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.476155 19591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:26.476225 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "96869395642d4b1fb27fe707692ff79f"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "af683006f86d2fad08b93683688a587e"
server_key_iv: "9666b284f749c864cc7cf826112b49c3"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.476279 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:26.484335 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:26.486023 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:26.486131 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:26.486299 19591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:26.486522 19591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:26.486549 19591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.486570 19591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:26.486582 19591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.492669 19591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44199
I20260924 00:13:26.492713 19719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44199 every 8 connection(s)
I20260924 00:13:26.493053 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:26.501159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19591
I20260924 00:13:26.501297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:26.501488 392 external_mini_cluster.cc:1428] Setting key 85421a2cd247058722931ca942a07254
I20260924 00:13:26.502521 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.493859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53013 (local address 127.0.98.62:45917)
0924 00:13:26.493948 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.493950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.494148 (+ 198us) server_negotiation.cc:407] Connection header received
0924 00:13:26.495047 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.495048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.495059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.495125 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:26.495801 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.496359 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.497318 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.497531 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.497588 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.499778 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:26.499779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:26.499779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:26.499786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:26.501052 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.501542 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.501544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.501544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.501611 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.501856 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.501856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.501857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.501985 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:26.502052 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.502376 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.502449 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:13:26.502835 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.493822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.1:53013)
0924 00:13:26.493983 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.494035 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.494942 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.495198 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.495201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.495529 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:26.495721 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.495725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.496443 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.496444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.497244 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.497246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.497606 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.497607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.497702 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.498262 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:26.498270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:26.499693 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:26.501144 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.501150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.501151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.501466 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.501685 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.501686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.501686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.501801 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.502069 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:26.502075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:26.502234 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.502355 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.502549 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260924 00:13:26.503413 19720 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260924 00:13:26.503536 19720 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:26.503717 19720 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
I20260924 00:13:26.504252 19529 ts_manager.cc:194] Registered new tserver with Master: 96869395642d4b1fb27fe707692ff79f (127.0.98.1:44199)
I20260924 00:13:26.505162 19529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53013
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:13:26.527060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:26.607098 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:26.607347 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:26.607385 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:26.608583 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:26.608652 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:26.608697 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:26.610193 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:26.610452 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:26.610682 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:26.615868 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.615933 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.615895 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:26.616017 19725 server_base.cc:1034] running on GCE node
I20260924 00:13:26.616366 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:26.616621 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:26.617811 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790208806617793 us; error 35 us; skew 500 ppm
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:26.620653 19725 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:13:26.621032 19725 webserver.cc:460] Webserver started at http://127.0.98.2:36363/ using document root <none> and password file <none>
I20260924 00:13:26.621214 19725 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:26.621253 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:26.621338 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:26.622356 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "1a5787f0a14b2b11a3ce2d901d18688e"
server_key_iv: "4614863004d9202f762573d35e5893f2"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.622656 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "1a5787f0a14b2b11a3ce2d901d18688e"
server_key_iv: "4614863004d9202f762573d35e5893f2"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.623942 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:26.624604 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.624788 19725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:26.624869 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "1a5787f0a14b2b11a3ce2d901d18688e"
server_key_iv: "4614863004d9202f762573d35e5893f2"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.624925 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:26.640303 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:26.642218 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:26.642326 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:26.642503 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:26.642742 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:26.642777 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.642798 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:26.642810 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.648373 19725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33191
I20260924 00:13:26.648469 19853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33191 every 8 connection(s)
I20260924 00:13:26.648777 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:26.651968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19725
I20260924 00:13:26.652092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:26.652258 392 external_mini_cluster.cc:1428] Setting key 307dadda8b61013b89e407ba373242a4
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:26.659296 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.649549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41511 (local address 127.0.98.62:45917)
0924 00:13:26.649629 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.649632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.649794 (+ 162us) server_negotiation.cc:407] Connection header received
0924 00:13:26.650913 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.650914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.650923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.650974 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:26.651620 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.652370 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.653610 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.653930 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.654008 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.656461 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:26.656462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:26.656463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:26.656471 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:26.657889 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.658306 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.658308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.658308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.658370 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.658622 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.658623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.658623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.658741 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:26.658817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.659098 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.659180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:13:26.659509 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.649498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.2:41511)
0924 00:13:26.649671 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.649709 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.650811 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.651034 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.651037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.651334 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:26.651545 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.651549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.652467 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.652468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.653520 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.653522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.654040 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.654041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.654192 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.654753 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:26.654764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:26.656359 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:26.657991 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.657998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.658001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.658229 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.658437 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.658438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.658438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.658553 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.658809 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:26.658815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:26.658960 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.659072 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.659256 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260924 00:13:26.660069 19854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260924 00:13:26.660184 19854 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:26.660382 19854 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:13:26.660750 19529 ts_manager.cc:194] Registered new tserver with Master: 7c76dfe4bb9445e78140b2c9fd73d5a1 (127.0.98.2:33191)
I20260924 00:13:26.661142 19529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:13:26.678921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:26.760396 19859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:26.760605 19859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:26.760629 19859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:26.761699 19859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:26.761734 19859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:26.761759 19859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:26.763262 19859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:26.763456 19859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:26.763628 19859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:26.768464 19864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.768474 19867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:26.768499 19865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:26.768766 19859 server_base.cc:1034] running on GCE node
I20260924 00:13:26.768985 19859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:26.769243 19859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:26.770401 19859 hybrid_clock.cc:648] HybridClock initialized: now 1790208806770397 us; error 36 us; skew 500 ppm
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:26.772958 19859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:13:26.773451 19859 webserver.cc:460] Webserver started at http://127.0.98.3:40187/ using document root <none> and password file <none>
I20260924 00:13:26.773690 19859 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:26.773756 19859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:26.773906 19859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:26.774948 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f31b6860e164b7a82a78fa98fe27dad"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "6c7774d1366c5e398ea7f82c1f21aa7d"
server_key_iv: "04d424c2d5f2f0c2cdf27ad45ea92209"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.775388 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f31b6860e164b7a82a78fa98fe27dad"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "6c7774d1366c5e398ea7f82c1f21aa7d"
server_key_iv: "04d424c2d5f2f0c2cdf27ad45ea92209"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.776685 19859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:26.777436 19874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.777619 19859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:26.777720 19859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "0f31b6860e164b7a82a78fa98fe27dad"
format_stamp: "Formatted at 2026-09-24 00:13:26 on dist-test-slave-3q8k"
server_key: "6c7774d1366c5e398ea7f82c1f21aa7d"
server_key_iv: "04d424c2d5f2f0c2cdf27ad45ea92209"
server_key_version: "encryptionkey@0"
I20260924 00:13:26.777806 19859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:26.794520 19859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:26.797051 19859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:26.797216 19859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:26.797451 19859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:26.797755 19859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:26.797811 19859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.797849 19859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:26.797878 19859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:26.803802 19859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42433
I20260924 00:13:26.803864 19987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42433 every 8 connection(s)
I20260924 00:13:26.804215 19859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:13:26 dist-test-slave-3q8k krb5kdc[19484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:26.813179 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.804999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35465 (local address 127.0.98.62:45917)
0924 00:13:26.805083 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.805085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.805261 (+ 176us) server_negotiation.cc:407] Connection header received
0924 00:13:26.806166 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.806167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.806176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.806235 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:26.806873 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.807378 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.808344 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.808519 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.808568 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.810538 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:26.810539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:26.810540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:26.810546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:26.811967 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.812418 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.812419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.812420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.812467 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.812674 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.812674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.812675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.812771 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:26.812818 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.813054 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.813114 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:13:26.813340 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.804948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.3:35465)
0924 00:13:26.805131 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.805174 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.806080 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.806290 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.806294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.806604 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:26.806795 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.806808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.807450 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.807451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.808268 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.808270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.808594 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.808595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.808693 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.809136 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:26.809144 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:26.810452 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:26.812044 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.812050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.812052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.812343 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.812524 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:26.812524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:26.812524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:26.812622 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:26.812825 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:26.812829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:26.812941 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.813028 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.813143 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260924 00:13:26.813934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 19859
I20260924 00:13:26.814020 19988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260924 00:13:26.814054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:26.814122 19988 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:26.814246 392 external_mini_cluster.cc:1428] Setting key 465d5efb1c467413a48dd206350b8057
I20260924 00:13:26.814329 19988 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
I20260924 00:13:26.814754 19529 ts_manager.cc:194] Registered new tserver with Master: 0f31b6860e164b7a82a78fa98fe27dad (127.0.98.3:42433)
I20260924 00:13:26.815136 19529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35465
I20260924 00:13:26.815150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:26.822610 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.816499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.0.98.62:45917)
0924 00:13:26.816562 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.816564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.816659 (+ 95us) server_negotiation.cc:407] Connection header received
0924 00:13:26.816722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.816723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.816733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.816775 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:26.817404 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.817867 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.818651 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.818851 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.818921 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.820217 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:26.820218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:26.820219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:26.820225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:26.821488 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.821825 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.821826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.821826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.821882 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:26.822086 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:26.822087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:26.822087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:26.822229 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:26.822337 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.822437 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.822512 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:13:26.824040 19529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:26.825645 19529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:26.832759 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.828025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44199 (local address 127.0.0.1:54574)
0924 00:13:26.828923 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.828933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.828988 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.829156 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.829157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.829159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:26.829205 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.829206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.830474 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.830475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.831516 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.831519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.832465 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.832466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.832540 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.832555 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.832624 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.832678 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":48,"threads_started":1}
I20260924 00:13:26.833039 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.828136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.98.1:44199)
0924 00:13:26.828321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.828326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.828985 (+ 659us) server_negotiation.cc:407] Connection header received
0924 00:13:26.829054 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.829056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.829071 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.829143 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:26.829323 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.830348 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.831651 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.832376 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.832466 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.832528 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.832645 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.832683 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260924 00:13:26.833273 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.827816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33191 (local address 127.0.0.1:43876)
0924 00:13:26.828105 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.828148 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.828213 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.828540 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.828543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.828546 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:26.828614 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.828615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.830015 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.830016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.832150 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.832153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.833037 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.833038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.833103 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.833108 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.833164 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.833201 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":783,"thread_start_us":63,"threads_started":1}
I20260924 00:13:26.833320 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.827926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.2:33191)
0924 00:13:26.828338 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.828343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.828352 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:26.828393 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.828395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.828407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.828503 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:26.828771 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.829901 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.832249 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.832961 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.833024 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.833074 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.833184 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.833230 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":82,"threads_started":1}
I20260924 00:13:26.833770 19654 tablet_service.cc:1467] Processing CreateTablet for tablet 347723228cf2419694e8e1a527433c7a (DEFAULT_TABLE table=test-table [id=ff6e62d31667434f8eb8fa002aecd913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:26.833885 19788 tablet_service.cc:1467] Processing CreateTablet for tablet 347723228cf2419694e8e1a527433c7a (DEFAULT_TABLE table=test-table [id=ff6e62d31667434f8eb8fa002aecd913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:26.834131 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347723228cf2419694e8e1a527433c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:26.834154 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347723228cf2419694e8e1a527433c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:26.837477 20002 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Bootstrap starting.
I20260924 00:13:26.837599 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.827914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.0.98.3:42433)
0924 00:13:26.828677 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.828681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.828690 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:26.828726 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.828728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.828743 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.828820 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:26.828922 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.829985 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.832062 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.832757 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.833048 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.833190 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:26.837468 (+ 4278us) server_negotiation.cc:299] Negotiation successful
0924 00:13:26.837517 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":124,"threads_started":1}
I20260924 00:13:26.837773 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.827832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42433 (local address 127.0.0.1:46662)
0924 00:13:26.828118 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.828148 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.828212 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.828815 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.828816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.828818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:26.828863 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.828865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.830078 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.830079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.831976 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.831979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.833229 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.833230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:26.833302 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:26.833306 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:26.833471 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 00:13:26.833509 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":769,"thread_start_us":100,"threads_started":1}
I20260924 00:13:26.838196 20002 tablet_bootstrap.cc:654] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:26.838485 19922 tablet_service.cc:1467] Processing CreateTablet for tablet 347723228cf2419694e8e1a527433c7a (DEFAULT_TABLE table=test-table [id=ff6e62d31667434f8eb8fa002aecd913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:26.838786 20002 log.cc:824] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:26.838822 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347723228cf2419694e8e1a527433c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:26.839263 20003 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Bootstrap starting.
I20260924 00:13:26.839876 20003 tablet_bootstrap.cc:654] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:26.840452 20003 log.cc:824] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:26.841018 20002 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: No bootstrap required, opened a new log
I20260924 00:13:26.841094 20002 ts_tablet_manager.cc:1397] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:13:26.841182 20003 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: No bootstrap required, opened a new log
I20260924 00:13:26.841252 20003 ts_tablet_manager.cc:1397] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:26.842204 20006 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Bootstrap starting.
I20260924 00:13:26.842870 20006 tablet_bootstrap.cc:654] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:26.842875 20002 raft_consensus.cc:348] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.843103 20002 raft_consensus.cc:374] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:26.843148 20002 raft_consensus.cc:729] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76dfe4bb9445e78140b2c9fd73d5a1, State: Initialized, Role: FOLLOWER
I20260924 00:13:26.843317 20002 consensus_queue.cc:260] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.843084 20003 raft_consensus.cc:348] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.843456 20003 raft_consensus.cc:374] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:26.843526 20003 raft_consensus.cc:729] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96869395642d4b1fb27fe707692ff79f, State: Initialized, Role: FOLLOWER
I20260924 00:13:26.843533 20006 log.cc:824] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:26.843609 20002 ts_tablet_manager.cc:1428] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 00:13:26.843668 20003 consensus_queue.cc:260] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.843758 19854 heartbeater.cc:503] Master 127.0.98.62:45917 was elected leader, sending a full tablet report...
I20260924 00:13:26.843984 20003 ts_tablet_manager.cc:1428] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:26.843997 19720 heartbeater.cc:503] Master 127.0.98.62:45917 was elected leader, sending a full tablet report...
I20260924 00:13:26.844378 20006 tablet_bootstrap.cc:492] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: No bootstrap required, opened a new log
I20260924 00:13:26.844452 20006 ts_tablet_manager.cc:1397] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:26.845949 20006 raft_consensus.cc:348] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.846124 20006 raft_consensus.cc:374] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:26.846174 20006 raft_consensus.cc:729] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f31b6860e164b7a82a78fa98fe27dad, State: Initialized, Role: FOLLOWER
I20260924 00:13:26.846289 20006 consensus_queue.cc:260] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.846532 20006 ts_tablet_manager.cc:1428] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:13:26.846575 19988 heartbeater.cc:503] Master 127.0.98.62:45917 was elected leader, sending a full tablet report...
W20260924 00:13:26.899441 19855 tablet.cc:2369] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:13:26.993793 19721 tablet.cc:2369] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:26.997490 20007 raft_consensus.cc:482] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:26.997625 20007 raft_consensus.cc:504] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:26.998059 20007 leader_election.cc:288] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f31b6860e164b7a82a78fa98fe27dad (127.0.98.3:42433), 96869395642d4b1fb27fe707692ff79f (127.0.98.1:44199)
I20260924 00:13:27.002038 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.998364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34505 (local address 127.0.98.1:44199)
0924 00:13:26.998407 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.998410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.998576 (+ 166us) server_negotiation.cc:407] Connection header received
0924 00:13:26.998603 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.998603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.998616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.998660 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:26.998824 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.999541 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001101 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.001675 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001737 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.001745 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.001930 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.001969 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:13:27.002074 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.998305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43175 (local address 127.0.98.3:42433)
0924 00:13:26.998375 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:26.998377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:26.998388 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:26.998417 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:26.998418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:26.998429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:26.998493 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:26.998726 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.999524 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001092 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.001651 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001733 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.001744 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.001930 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.002008 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:13:27.002059 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.998215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42433 (local address 127.0.98.2:43175)
0924 00:13:26.998279 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.998291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.998343 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.998531 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.998532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.998534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:26.998618 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.998620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.999619 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.999620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.000968 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.000971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.001757 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.001851 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.001853 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.001907 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.001955 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":103}
I20260924 00:13:27.002069 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:26.998268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44199 (local address 127.0.98.2:34505)
0924 00:13:26.998451 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:26.998467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:26.998547 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:26.998687 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:26.998688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:26.998690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:26.998745 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:26.998746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:26.999619 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:26.999620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.000968 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.000971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.001757 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.001758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.001840 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.001844 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.001907 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.001955 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":61,"thread_start_us":114,"threads_started":1}
I20260924 00:13:27.002450 19942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347723228cf2419694e8e1a527433c7a" candidate_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f31b6860e164b7a82a78fa98fe27dad" is_pre_election: true
I20260924 00:13:27.002501 19674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347723228cf2419694e8e1a527433c7a" candidate_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96869395642d4b1fb27fe707692ff79f" is_pre_election: true
I20260924 00:13:27.002630 19674 raft_consensus.cc:2454] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76dfe4bb9445e78140b2c9fd73d5a1 in term 0.
I20260924 00:13:27.002631 19942 raft_consensus.cc:2454] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c76dfe4bb9445e78140b2c9fd73d5a1 in term 0.
I20260924 00:13:27.002879 19742 leader_election.cc:302] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c76dfe4bb9445e78140b2c9fd73d5a1, 96869395642d4b1fb27fe707692ff79f; no voters:
I20260924 00:13:27.003078 20007 raft_consensus.cc:2781] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:27.003159 20007 raft_consensus.cc:482] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:27.003191 20007 raft_consensus.cc:3037] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.003893 20007 raft_consensus.cc:504] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:27.004045 20007 leader_election.cc:288] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [CANDIDATE]: Term 1 election: Requested vote from peers 0f31b6860e164b7a82a78fa98fe27dad (127.0.98.3:42433), 96869395642d4b1fb27fe707692ff79f (127.0.98.1:44199)
I20260924 00:13:27.004223 19942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347723228cf2419694e8e1a527433c7a" candidate_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f31b6860e164b7a82a78fa98fe27dad"
I20260924 00:13:27.004226 19674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347723228cf2419694e8e1a527433c7a" candidate_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96869395642d4b1fb27fe707692ff79f"
I20260924 00:13:27.004299 19942 raft_consensus.cc:3037] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.004323 19674 raft_consensus.cc:3037] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.005024 19674 raft_consensus.cc:2454] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76dfe4bb9445e78140b2c9fd73d5a1 in term 1.
I20260924 00:13:27.005033 19942 raft_consensus.cc:2454] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c76dfe4bb9445e78140b2c9fd73d5a1 in term 1.
I20260924 00:13:27.005203 19741 leader_election.cc:302] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f31b6860e164b7a82a78fa98fe27dad, 7c76dfe4bb9445e78140b2c9fd73d5a1; no voters:
I20260924 00:13:27.005311 20007 raft_consensus.cc:2781] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:27.005483 20007 raft_consensus.cc:686] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c76dfe4bb9445e78140b2c9fd73d5a1, State: Running, Role: LEADER
I20260924 00:13:27.005612 20007 consensus_queue.cc:237] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } }
I20260924 00:13:27.006517 19528 catalog_manager.cc:5495] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c76dfe4bb9445e78140b2c9fd73d5a1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c76dfe4bb9445e78140b2c9fd73d5a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20260924 00:13:27.012571 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.010126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.2:33191)
0924 00:13:27.010174 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.010176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.010184 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:27.010219 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.010219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.010237 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.010307 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:27.010422 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.010890 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.011758 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.011978 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.012063 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.012223 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:27.012331 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:27.012419 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.012441 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.012460 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:13:27.014279 19942 raft_consensus.cc:1261] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 1 FOLLOWER]: Refusing update from remote peer 7c76dfe4bb9445e78140b2c9fd73d5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:27.014286 19674 raft_consensus.cc:1261] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 1 FOLLOWER]: Refusing update from remote peer 7c76dfe4bb9445e78140b2c9fd73d5a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:27.014632 20007 consensus_queue.cc:1035] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96869395642d4b1fb27fe707692ff79f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:27.014761 20012 consensus_queue.cc:1035] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f31b6860e164b7a82a78fa98fe27dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:27.019124 19528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:27.019238 19528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:27.020030 19528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dab30be9b704d1ba6926608329d2d48: Sending DeleteTablet for 3 replicas of tablet 347723228cf2419694e8e1a527433c7a
I20260924 00:13:27.020309 19921 tablet_service.cc:1514] Processing DeleteTablet for tablet 347723228cf2419694e8e1a527433c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46662
I20260924 00:13:27.020412 19654 tablet_service.cc:1514] Processing DeleteTablet for tablet 347723228cf2419694e8e1a527433c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54574
I20260924 00:13:27.020594 20024 tablet_replica.cc:330] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: stopping tablet replica
I20260924 00:13:27.020310 19922 tablet_service.cc:1514] Processing DeleteTablet for tablet 347723228cf2419694e8e1a527433c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46662
I20260924 00:13:27.020648 20025 tablet_replica.cc:330] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: stopping tablet replica
I20260924 00:13:27.020751 20025 raft_consensus.cc:2229] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:27.020778 20024 raft_consensus.cc:2229] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:27.020865 20025 raft_consensus.cc:2258] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:27.020974 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet 347723228cf2419694e8e1a527433c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43876
I20260924 00:13:27.021234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19591
I20260924 00:13:27.021291 20025 ts_tablet_manager.cc:1905] T 347723228cf2419694e8e1a527433c7a P 96869395642d4b1fb27fe707692ff79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:27.023999 20024 raft_consensus.cc:2258] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:27.024436 20024 ts_tablet_manager.cc:1905] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:27.026407 20026 tablet_replica.cc:330] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: stopping tablet replica
I20260924 00:13:27.026454 20024 ts_tablet_manager.cc:1918] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:27.026518 20024 log.cc:1196] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/347723228cf2419694e8e1a527433c7a
I20260924 00:13:27.026522 20026 raft_consensus.cc:2229] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:27.026612 20026 raft_consensus.cc:2258] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:27.026801 20024 ts_tablet_manager.cc:1939] T 347723228cf2419694e8e1a527433c7a P 0f31b6860e164b7a82a78fa98fe27dad: Deleting consensus metadata
I20260924 00:13:27.027092 20026 ts_tablet_manager.cc:1905] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:27.027635 19513 catalog_manager.cc:4841] TS 0f31b6860e164b7a82a78fa98fe27dad (127.0.98.3:42433): tablet 347723228cf2419694e8e1a527433c7a (table test-table [id=ff6e62d31667434f8eb8fa002aecd913]) successfully deleted
W20260924 00:13:27.027779 19513 catalog_manager.cc:4808] TS 0f31b6860e164b7a82a78fa98fe27dad (127.0.98.3:42433): delete failed for tablet 347723228cf2419694e8e1a527433c7a because the tablet was not found. No further retry: Not found: Tablet not found: 347723228cf2419694e8e1a527433c7a
W20260924 00:13:27.028148 19514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44199 (error 108)
I20260924 00:13:27.028607 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.028465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44199 (local address 127.0.0.1:54588)
0924 00:13:27.028526 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.028575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 00:13:27.028710 19514 catalog_manager.cc:4584] TS 96869395642d4b1fb27fe707692ff79f (127.0.98.1:44199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 347723228cf2419694e8e1a527433c7a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44199: connect: Connection refused (error 111)
I20260924 00:13:27.028978 20026 ts_tablet_manager.cc:1918] T 347723228cf2419694e8e1a527433c7a P 7c76dfe4bb9445e78140b2c9fd73d5a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:27.028980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19725
I20260924 00:13:27.034123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19859
I20260924 00:13:27.034199 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.034054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33191 (local address 127.0.0.1:43902)
0924 00:13:27.034112 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.034141 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:13:27.038673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 19497
2026-09-24T00:13:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T00:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:27Z Disabled control of system clock
I20260924 00:13:27.065876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41189
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:27.148574 20031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:27.148824 20031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:27.148847 20031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:27.150099 20031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:27.150146 20031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:27.150159 20031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:27.150172 20031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:27.150184 20031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:27.151638 20031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:27.151895 20031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:27.152180 20031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:27.157426 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.157414 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.157420 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:27.157696 20031 server_base.cc:1034] running on GCE node
I20260924 00:13:27.157958 20031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:27.158281 20031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:27.159449 20031 hybrid_clock.cc:648] HybridClock initialized: now 1790208807159433 us; error 50 us; skew 500 ppm
I20260924 00:13:27.160835 20031 webserver.cc:460] Webserver started at http://127.0.98.62:43903/ using document root <none> and password file <none>
I20260924 00:13:27.161129 20031 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:27.161206 20031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:27.161365 20031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:27.162487 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "33c4c87bb444433daeac712dd6ee11f5"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "c609fbbafd8e170751575a965d583240"
server_key_iv: "92993921de51f5b6846b336ee5ddd881"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.162854 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "33c4c87bb444433daeac712dd6ee11f5"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "c609fbbafd8e170751575a965d583240"
server_key_iv: "92993921de51f5b6846b336ee5ddd881"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.164278 20031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:27.165000 20045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.165184 20031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:27.165292 20031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "33c4c87bb444433daeac712dd6ee11f5"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "c609fbbafd8e170751575a965d583240"
server_key_iv: "92993921de51f5b6846b336ee5ddd881"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.165369 20031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:27.181264 20031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:27.181625 20031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:27.181788 20031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:27.186012 20031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41189
I20260924 00:13:27.186082 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41189 every 8 connection(s)
I20260924 00:13:27.186457 20031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:27.187455 20098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:27.189805 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: Bootstrap starting.
I20260924 00:13:27.190415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20031
I20260924 00:13:27.190436 20098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:27.190510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:27.190712 392 external_mini_cluster.cc:1428] Setting key ec23d190d7a43d2d7b7d70bc7772186a
I20260924 00:13:27.190778 20098 log.cc:824] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:27.191656 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: No bootstrap required, opened a new log
I20260924 00:13:27.193596 20098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260924 00:13:27.193766 20098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:27.193815 20098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c4c87bb444433daeac712dd6ee11f5, State: Initialized, Role: FOLLOWER
I20260924 00:13:27.193958 20098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260924 00:13:27.194046 20098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:27.194089 20098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:27.194139 20098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.194702 20098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260924 00:13:27.194829 20098 leader_election.cc:302] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c4c87bb444433daeac712dd6ee11f5; no voters:
I20260924 00:13:27.195072 20098 leader_election.cc:288] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:27.195077 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:27.195300 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [term 1 LEADER]: Becoming Leader. State: Replica: 33c4c87bb444433daeac712dd6ee11f5, State: Running, Role: LEADER
I20260924 00:13:27.195389 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260924 00:13:27.195719 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c4c87bb444433daeac712dd6ee11f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } } }
I20260924 00:13:27.195807 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:27.195833 20098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:27.196139 20107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:27.196691 20107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:27.196774 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.191266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.98.62:41189)
0924 00:13:27.191473 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.191482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.191514 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:13:27.192004 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.192012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.192073 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.192223 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:27.192454 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.193350 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.194370 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.194790 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.194967 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.195832 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:27.195837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:27.195837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:27.196053 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:27.196099 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:27.196115 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:27.196181 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.196450 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.196546 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 00:13:27.195708 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c4c87bb444433daeac712dd6ee11f5. Latest consensus state: current_term: 1 leader_uuid: "33c4c87bb444433daeac712dd6ee11f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4c87bb444433daeac712dd6ee11f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } } }
I20260924 00:13:27.196916 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:27.198287 20107 catalog_manager.cc:1347] Generated new cluster ID: d7c5a7039602464f89f03b19cd2183dc
I20260924 00:13:27.198338 20107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:27.211413 20107 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:27.211959 20107 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:27.214845 20107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: Generated new TSK 0
I20260924 00:13:27.215022 20107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:27.217478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:27.298519 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:27.298717 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:27.298739 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:27.299973 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:27.300014 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:27.300040 20122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:13:27.301363 20122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:27.301605 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:27.301934 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:27.307670 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.307678 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.307865 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:27.307825 20122 server_base.cc:1034] running on GCE node
I20260924 00:13:27.308079 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:27.308328 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:27.309479 20122 hybrid_clock.cc:648] HybridClock initialized: now 1790208807309466 us; error 26 us; skew 500 ppm
I20260924 00:13:27.310750 20122 webserver.cc:460] Webserver started at http://127.0.98.1:43791/ using document root <none> and password file <none>
I20260924 00:13:27.311028 20122 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:27.311075 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:27.311203 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:27.312340 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data/instance:
uuid: "f780c66bd3584c69a4cb96f8b82ec853"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "b75a3339d88954e3e03bc4e247c4912b"
server_key_iv: "f61ce385b49735e6ef8552f4b00ba192"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.312686 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f780c66bd3584c69a4cb96f8b82ec853"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "b75a3339d88954e3e03bc4e247c4912b"
server_key_iv: "f61ce385b49735e6ef8552f4b00ba192"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.313974 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:13:27.314725 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.314877 20122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:27.314987 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal
uuid: "f780c66bd3584c69a4cb96f8b82ec853"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "b75a3339d88954e3e03bc4e247c4912b"
server_key_iv: "f61ce385b49735e6ef8552f4b00ba192"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.315062 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:27.328537 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:27.328832 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:27.328965 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:27.329203 20122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:27.329523 20122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:27.329576 20122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.329615 20122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:27.329644 20122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.335568 20122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41681
I20260924 00:13:27.335602 20249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41681 every 8 connection(s)
I20260924 00:13:27.335947 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:13:27.341221 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.336720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37169 (local address 127.0.98.62:41189)
0924 00:13:27.336763 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.336765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.336973 (+ 208us) server_negotiation.cc:407] Connection header received
0924 00:13:27.337535 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.337536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.337550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.337599 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:27.338266 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.338777 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.339847 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.340057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.340144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.340900 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:27.340901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:27.340902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:27.340907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:27.340920 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:27.340924 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:27.340976 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.341091 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.341121 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:27.341441 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.336685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.1:37169)
0924 00:13:27.336856 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.336890 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.337414 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.337666 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.337670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.337680 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:27.338175 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.338179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.338873 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.338874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.339744 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.339746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.340154 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.340155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.340285 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.340764 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:27.340765 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:27.340792 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:27.340814 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:27.341012 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:27.341016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:27.341019 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.341093 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.341207 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260924 00:13:27.341993 20250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260924 00:13:27.342072 20250 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:27.342361 20250 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260924 00:13:27.342417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20122
I20260924 00:13:27.342497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:13:27.342718 392 external_mini_cluster.cc:1428] Setting key 9d701913f2a37ec9ca11eec86deebb01
I20260924 00:13:27.343061 20062 ts_manager.cc:194] Registered new tserver with Master: f780c66bd3584c69a4cb96f8b82ec853 (127.0.98.1:41681)
I20260924 00:13:27.343853 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37169
I20260924 00:13:27.343871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:13:27.425673 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:27.425868 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:27.425889 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:27.427045 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:27.427086 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:27.427111 20253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:13:27.428481 20253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:27.428725 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:27.429018 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:27.434128 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:27.434281 20253 server_base.cc:1034] running on GCE node
W20260924 00:13:27.434158 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.434415 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:27.434677 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:27.434875 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:27.436012 20253 hybrid_clock.cc:648] HybridClock initialized: now 1790208807435990 us; error 37 us; skew 500 ppm
I20260924 00:13:27.437333 20253 webserver.cc:460] Webserver started at http://127.0.98.2:38883/ using document root <none> and password file <none>
I20260924 00:13:27.437600 20253 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:27.437674 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:27.437829 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:27.438903 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data/instance:
uuid: "509ec62c26c04a799997cfc3ae530abd"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "9e8a65c682b65db24b165365a3f1fa35"
server_key_iv: "b00545bac4d5422591ac64fd4acc1220"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.439332 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal/instance:
uuid: "509ec62c26c04a799997cfc3ae530abd"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "9e8a65c682b65db24b165365a3f1fa35"
server_key_iv: "b00545bac4d5422591ac64fd4acc1220"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.440622 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:27.441481 20267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.441774 20253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:27.441874 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal
uuid: "509ec62c26c04a799997cfc3ae530abd"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "9e8a65c682b65db24b165365a3f1fa35"
server_key_iv: "b00545bac4d5422591ac64fd4acc1220"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.441999 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:27.456030 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:27.456347 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:27.456506 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:27.456743 20253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:27.457083 20253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:27.457139 20253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.457181 20253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:27.457211 20253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.463497 20253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37803
I20260924 00:13:27.463558 20380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37803 every 8 connection(s)
I20260924 00:13:27.463891 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:13:27.468564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20253
I20260924 00:13:27.468698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:13:27.469024 392 external_mini_cluster.cc:1428] Setting key b4a04feca89c7798613c794f89dbd01f
I20260924 00:13:27.469408 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.464640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36919 (local address 127.0.98.62:41189)
0924 00:13:27.464685 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.464688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.464926 (+ 238us) server_negotiation.cc:407] Connection header received
0924 00:13:27.465472 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.465473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.465484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.465540 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:27.466180 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.466697 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.467778 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.468052 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.468117 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.469148 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:27.469150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:27.469151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:27.469155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:27.469171 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:27.469175 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:27.469225 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.469300 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.469329 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:13:27.469631 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.464652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.2:36919)
0924 00:13:27.464825 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.464858 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.465326 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.465594 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.465598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.465606 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:27.466103 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.466107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.466778 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.466779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.467658 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.467660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.468132 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.468133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.468247 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.468982 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:27.468983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:27.469023 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:27.469070 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:27.469225 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:27.469231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:27.469234 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.469295 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.469398 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260924 00:13:27.470165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--builtin_ntp_servers=127.0.98.20:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:13:27.470206 20381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260924 00:13:27.470294 20381 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:27.470494 20381 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260924 00:13:27.470855 20062 ts_manager.cc:194] Registered new tserver with Master: 509ec62c26c04a799997cfc3ae530abd (127.0.98.2:37803)
I20260924 00:13:27.471282 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36919
W20260924 00:13:27.551875 20384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:27.552127 20384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:27.552165 20384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:27.553310 20384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:27.553380 20384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:27.553422 20384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:13:27.554890 20384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:27.555184 20384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:27.555540 20384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:27.560765 20390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.560794 20392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:27.560765 20389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:27.561082 20384 server_base.cc:1034] running on GCE node
I20260924 00:13:27.561331 20384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:27.561586 20384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:27.562789 20384 hybrid_clock.cc:648] HybridClock initialized: now 1790208807562765 us; error 34 us; skew 500 ppm
I20260924 00:13:27.564394 20384 webserver.cc:460] Webserver started at http://127.0.98.3:43993/ using document root <none> and password file <none>
I20260924 00:13:27.564677 20384 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:27.564759 20384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:27.564911 20384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:27.566013 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data/instance:
uuid: "37bd7ff7d5e344f285a9b8648883a87c"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "369cb10ea202b3c6681602a79b0a4d21"
server_key_iv: "81ca7812d6fa05c41fc7b4f7879ca6fd"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.566375 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37bd7ff7d5e344f285a9b8648883a87c"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "369cb10ea202b3c6681602a79b0a4d21"
server_key_iv: "81ca7812d6fa05c41fc7b4f7879ca6fd"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.567807 20384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:27.568689 20398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.569015 20384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:27.569115 20384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal
uuid: "37bd7ff7d5e344f285a9b8648883a87c"
format_stamp: "Formatted at 2026-09-24 00:13:27 on dist-test-slave-3q8k"
server_key: "369cb10ea202b3c6681602a79b0a4d21"
server_key_iv: "81ca7812d6fa05c41fc7b4f7879ca6fd"
server_key_version: "encryptionkey@0"
I20260924 00:13:27.569226 20384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:27.577260 20384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:27.577569 20384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:27.577719 20384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:27.577951 20384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:13:27.578277 20384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:13:27.578332 20384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.578364 20384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:13:27.578390 20384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:27.584231 20384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44729
I20260924 00:13:27.584271 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44729 every 8 connection(s)
I20260924 00:13:27.584627 20384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:13:27.585240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20384
I20260924 00:13:27.585364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:13:27.585562 392 external_mini_cluster.cc:1428] Setting key 1cb69b24882899ec423c288db120670b
I20260924 00:13:27.590054 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.585460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54817 (local address 127.0.98.62:41189)
0924 00:13:27.585521 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.585524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.585828 (+ 304us) server_negotiation.cc:407] Connection header received
0924 00:13:27.586529 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.586530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.586543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.586647 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:27.587359 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.587840 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.588831 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.589029 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.589083 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.589836 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:27.589837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:27.589838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:27.589842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:27.589856 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:27.589859 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:27.589902 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.589974 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.589993 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:13:27.590298 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.585482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.3:54817)
0924 00:13:27.585676 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.585724 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.586436 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.586712 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.586715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.586724 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:27.587280 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.587286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.587978 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.587979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.588753 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.588755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.589100 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.589101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.589211 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.589705 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:13:27.589706 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:27.589733 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:13:27.589756 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:27.589914 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:27.589918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:27.589920 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.589971 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.590046 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260924 00:13:27.590865 20512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260924 00:13:27.590975 20512 heartbeater.cc:464] Registering TS with master...
I20260924 00:13:27.591163 20512 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260924 00:13:27.591552 20062 ts_manager.cc:194] Registered new tserver with Master: 37bd7ff7d5e344f285a9b8648883a87c (127.0.98.3:44729)
I20260924 00:13:27.591943 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54817
I20260924 00:13:27.597508 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:13:27.602119 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.598996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.98.62:41189)
0924 00:13:27.599057 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.599059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.599136 (+ 77us) server_negotiation.cc:407] Connection header received
0924 00:13:27.599249 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.599250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.599262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.599336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:27.599534 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.600077 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.600868 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.601055 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.601149 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.601834 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:27.601836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:27.601836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:27.601841 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:27.601865 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:27.601869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:27.601929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.601992 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.602021 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:13:27.603833 20062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:13:27.605450 20062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:13:27.611917 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.607659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37803 (local address 127.0.0.1:56668)
0924 00:13:27.607814 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.607853 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.607919 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.608468 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.608469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.608471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:27.608511 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.608512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609764 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.609765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.610668 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.610671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.611578 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.611580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.611672 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.611686 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.611766 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.611836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":53,"threads_started":1}
I20260924 00:13:27.612037 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.607671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:55364)
0924 00:13:27.608096 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.608105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.608142 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.608565 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.608566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.608567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:27.608613 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.608615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609506 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.609507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.610876 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.610878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.611757 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.611759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.611841 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.611846 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.611909 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.611951 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":39,"thread_start_us":89,"threads_started":1}
I20260924 00:13:27.612282 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.607962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.0.98.2:37803)
0924 00:13:27.608290 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.608295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.608304 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:27.608350 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.608353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.608369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.608579 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:27.608640 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609681 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.610763 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.611492 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.611568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.611658 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.612151 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.612192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260924 00:13:27.612473 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.608158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.0.98.1:41681)
0924 00:13:27.608332 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.608337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.608346 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:27.608394 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.608397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.608412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.608688 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:27.608738 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609416 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.610963 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.611692 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.611756 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.611857 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.611924 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.611962 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260924 00:13:27.612485 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.607660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44729 (local address 127.0.0.1:54732)
0924 00:13:27.607987 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.607998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.608050 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.608278 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.608281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.608282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:27.608349 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.608350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609638 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.609639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.610872 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.610875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.612229 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.612230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.612295 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.612298 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.612379 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.612417 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":29,"thread_start_us":115,"threads_started":1}
I20260924 00:13:27.612725 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.607914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.98.3:44729)
0924 00:13:27.608088 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.608092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.608102 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:13:27.608143 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.608145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.608161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.608247 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:27.608441 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.609558 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.611310 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.612054 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.612126 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.612209 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.612473 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.612514 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260924 00:13:27.613124 20314 tablet_service.cc:1467] Processing CreateTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 (DEFAULT_TABLE table=test-table [id=dcf00348977047c093a19a8ee5c3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:27.613214 20184 tablet_service.cc:1467] Processing CreateTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 (DEFAULT_TABLE table=test-table [id=dcf00348977047c093a19a8ee5c3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:27.613379 20446 tablet_service.cc:1467] Processing CreateTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 (DEFAULT_TABLE table=test-table [id=dcf00348977047c093a19a8ee5c3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:13:27.613533 20314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e16cf918b948ad8b8fb22cbadf0377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:27.613571 20184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e16cf918b948ad8b8fb22cbadf0377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:27.613663 20446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e16cf918b948ad8b8fb22cbadf0377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:27.616209 20526 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Bootstrap starting.
I20260924 00:13:27.616860 20527 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Bootstrap starting.
I20260924 00:13:27.617131 20526 tablet_bootstrap.cc:654] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:27.617475 20526 log.cc:824] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:27.617555 20527 tablet_bootstrap.cc:654] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:27.617849 20527 log.cc:824] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:27.618201 20528 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: Bootstrap starting.
I20260924 00:13:27.618520 20526 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: No bootstrap required, opened a new log
I20260924 00:13:27.618520 20527 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: No bootstrap required, opened a new log
I20260924 00:13:27.618593 20527 ts_tablet_manager.cc:1397] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:13:27.618593 20526 ts_tablet_manager.cc:1397] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:13:27.618827 20528 tablet_bootstrap.cc:654] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:27.619105 20528 log.cc:824] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:27.619824 20528 tablet_bootstrap.cc:492] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: No bootstrap required, opened a new log
I20260924 00:13:27.619910 20528 ts_tablet_manager.cc:1397] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:13:27.621260 20527 raft_consensus.cc:348] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.621357 20526 raft_consensus.cc:348] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.621447 20527 raft_consensus.cc:374] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:27.621483 20526 raft_consensus.cc:374] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:27.621483 20527 raft_consensus.cc:729] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bd7ff7d5e344f285a9b8648883a87c, State: Initialized, Role: FOLLOWER
I20260924 00:13:27.621521 20526 raft_consensus.cc:729] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509ec62c26c04a799997cfc3ae530abd, State: Initialized, Role: FOLLOWER
I20260924 00:13:27.621639 20526 consensus_queue.cc:260] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.621626 20527 consensus_queue.cc:260] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.621894 20526 ts_tablet_manager.cc:1428] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 00:13:27.621891 20527 ts_tablet_manager.cc:1428] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 00:13:27.622008 20512 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
I20260924 00:13:27.622020 20381 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
I20260924 00:13:27.622442 20528 raft_consensus.cc:348] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.622601 20528 raft_consensus.cc:374] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:27.622644 20528 raft_consensus.cc:729] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f780c66bd3584c69a4cb96f8b82ec853, State: Initialized, Role: FOLLOWER
I20260924 00:13:27.622754 20528 consensus_queue.cc:260] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.623011 20528 ts_tablet_manager.cc:1428] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:13:27.623155 20250 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
W20260924 00:13:27.714556 20382 tablet.cc:2369] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:13:27.734205 20533 raft_consensus.cc:482] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:13:27.734339 20533 raft_consensus.cc:504] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.734699 20533 leader_election.cc:288] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 509ec62c26c04a799997cfc3ae530abd (127.0.98.2:37803), f780c66bd3584c69a4cb96f8b82ec853 (127.0.98.1:41681)
I20260924 00:13:27.738626 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.734968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49965 (local address 127.0.98.2:37803)
0924 00:13:27.735022 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.735025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.735061 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:13:27.735096 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.735097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.735106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.735150 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:27.735335 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.736122 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.737692 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.738245 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.738323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.738332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.738513 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.738554 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:13:27.738626 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.735032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45031 (local address 127.0.98.1:41681)
0924 00:13:27.735073 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.735076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.735273 (+ 197us) server_negotiation.cc:407] Connection header received
0924 00:13:27.735303 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.735304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.735318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.735376 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:13:27.735492 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.736171 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.737692 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.738245 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.738323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.738332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.738513 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.738554 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:13:27.738634 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.734883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37803 (local address 127.0.98.3:49965)
0924 00:13:27.734951 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.734966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.735018 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.735183 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.735184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.735185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:27.735277 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.735279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.736230 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.736231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.737589 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.737592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.738339 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.738340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.738417 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.738421 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.738478 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.738522 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":62}
I20260924 00:13:27.738646 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.734885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.98.3:45031)
0924 00:13:27.735156 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.735166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:27.735238 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:27.735377 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:27.735377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:27.735379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:13:27.735427 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.735428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.736248 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.736249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.737591 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:27.737592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.738339 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.738340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:27.738417 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.738421 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:13:27.738478 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:13:27.738522 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":178,"thread_start_us":108,"threads_started":1}
I20260924 00:13:27.739085 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e16cf918b948ad8b8fb22cbadf0377" candidate_uuid: "37bd7ff7d5e344f285a9b8648883a87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f780c66bd3584c69a4cb96f8b82ec853" is_pre_election: true
I20260924 00:13:27.739504 20204 raft_consensus.cc:2454] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd7ff7d5e344f285a9b8648883a87c in term 0.
I20260924 00:13:27.739629 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e16cf918b948ad8b8fb22cbadf0377" candidate_uuid: "37bd7ff7d5e344f285a9b8648883a87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ec62c26c04a799997cfc3ae530abd" is_pre_election: true
I20260924 00:13:27.739795 20335 raft_consensus.cc:2454] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bd7ff7d5e344f285a9b8648883a87c in term 0.
I20260924 00:13:27.739996 20400 leader_election.cc:302] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd7ff7d5e344f285a9b8648883a87c, f780c66bd3584c69a4cb96f8b82ec853; no voters:
I20260924 00:13:27.740290 20533 raft_consensus.cc:2781] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:13:27.740355 20533 raft_consensus.cc:482] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:13:27.740372 20533 raft_consensus.cc:3037] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.741068 20533 raft_consensus.cc:504] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.741217 20533 leader_election.cc:288] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [CANDIDATE]: Term 1 election: Requested vote from peers 509ec62c26c04a799997cfc3ae530abd (127.0.98.2:37803), f780c66bd3584c69a4cb96f8b82ec853 (127.0.98.1:41681)
I20260924 00:13:27.741371 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e16cf918b948ad8b8fb22cbadf0377" candidate_uuid: "37bd7ff7d5e344f285a9b8648883a87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509ec62c26c04a799997cfc3ae530abd"
I20260924 00:13:27.741401 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e16cf918b948ad8b8fb22cbadf0377" candidate_uuid: "37bd7ff7d5e344f285a9b8648883a87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f780c66bd3584c69a4cb96f8b82ec853"
I20260924 00:13:27.741458 20335 raft_consensus.cc:3037] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.741470 20204 raft_consensus.cc:3037] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:27.742355 20204 raft_consensus.cc:2454] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd7ff7d5e344f285a9b8648883a87c in term 1.
I20260924 00:13:27.742355 20335 raft_consensus.cc:2454] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bd7ff7d5e344f285a9b8648883a87c in term 1.
I20260924 00:13:27.742544 20401 leader_election.cc:302] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bd7ff7d5e344f285a9b8648883a87c, 509ec62c26c04a799997cfc3ae530abd; no voters:
I20260924 00:13:27.742622 20533 raft_consensus.cc:2781] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:27.742780 20533 raft_consensus.cc:686] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 1 LEADER]: Becoming Leader. State: Replica: 37bd7ff7d5e344f285a9b8648883a87c, State: Running, Role: LEADER
I20260924 00:13:27.742904 20533 consensus_queue.cc:237] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } }
I20260924 00:13:27.743800 20062 catalog_manager.cc:5495] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bd7ff7d5e344f285a9b8648883a87c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "37bd7ff7d5e344f285a9b8648883a87c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bd7ff7d5e344f285a9b8648883a87c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 } health_report { overall_health: UNKNOWN } } }
I20260924 00:13:27.756930 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.754563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.0.98.3:44729)
0924 00:13:27.754629 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:27.754636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:27.754642 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:13:27.754673 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:27.754674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:27.754689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:27.754769 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:27.754874 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.755404 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.756166 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:27.756369 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:27.756461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:27.756595 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:27.756698 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:27.756791 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:27.756810 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:13:27.756826 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:13:27.758615 20335 raft_consensus.cc:1261] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 1 FOLLOWER]: Refusing update from remote peer 37bd7ff7d5e344f285a9b8648883a87c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:27.758615 20204 raft_consensus.cc:1261] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853 [term 1 FOLLOWER]: Refusing update from remote peer 37bd7ff7d5e344f285a9b8648883a87c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:27.758965 20536 consensus_queue.cc:1035] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f780c66bd3584c69a4cb96f8b82ec853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:27.759071 20533 consensus_queue.cc:1035] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [LEADER]: Connected to new peer: Peer: permanent_uuid: "509ec62c26c04a799997cfc3ae530abd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:27.763700 20061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:27.763801 20061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:13:27.764556 20061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c4c87bb444433daeac712dd6ee11f5: Sending DeleteTablet for 3 replicas of tablet a5e16cf918b948ad8b8fb22cbadf0377
I20260924 00:13:27.764918 20184 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='slave'} at 127.0.0.1:55364
I20260924 00:13:27.765012 20446 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='slave'} at 127.0.0.1:54732
I20260924 00:13:27.765067 20314 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e16cf918b948ad8b8fb22cbadf0377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:13:27 UTC) from {username='slave'} at 127.0.0.1:56668
I20260924 00:13:27.765261 20550 tablet_replica.cc:330] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: stopping tablet replica
I20260924 00:13:27.765269 20548 tablet_replica.cc:330] T a5e16cf918b948ad8b8fb22cbadf0377 P f780c66bd3584c69a4cb96f8b82ec853: stopping tablet replica
I20260924 00:13:27.765362 20550 raft_consensus.cc:2229] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:13:27.765385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20122
I20260924 00:13:27.766017 20550 raft_consensus.cc:2258] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:13:27.766441 20550 ts_tablet_manager.cc:1905] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:27.768129 20550 ts_tablet_manager.cc:1918] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:27.768194 20550 log.cc:1196] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-1/wal/wals/a5e16cf918b948ad8b8fb22cbadf0377
I20260924 00:13:27.768457 20550 ts_tablet_manager.cc:1939] T a5e16cf918b948ad8b8fb22cbadf0377 P 509ec62c26c04a799997cfc3ae530abd: Deleting consensus metadata
I20260924 00:13:27.768850 20048 catalog_manager.cc:4841] TS 509ec62c26c04a799997cfc3ae530abd (127.0.98.2:37803): tablet a5e16cf918b948ad8b8fb22cbadf0377 (table test-table [id=dcf00348977047c093a19a8ee5c3a8c4]) successfully deleted
I20260924 00:13:27.771379 20549 tablet_replica.cc:330] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: stopping tablet replica
I20260924 00:13:27.771483 20549 raft_consensus.cc:2229] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:13:27.771590 20549 raft_consensus.cc:2258] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:13:27.771797 20047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41681 (error 108)
I20260924 00:13:27.771981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20253
I20260924 00:13:27.772840 20549 ts_tablet_manager.cc:1905] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:13:27.772907 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:27.772116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41681 (local address 127.0.0.1:55366)
0924 00:13:27.772820 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:27.772873 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666}
I20260924 00:13:27.774272 20549 ts_tablet_manager.cc:1918] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:13:27.774360 20549 log.cc:1196] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Deleting WAL directory at /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790208568194258-392-0/minicluster-data/ts-2/wal/wals/a5e16cf918b948ad8b8fb22cbadf0377
I20260924 00:13:27.774592 20549 ts_tablet_manager.cc:1939] T a5e16cf918b948ad8b8fb22cbadf0377 P 37bd7ff7d5e344f285a9b8648883a87c: Deleting consensus metadata
I20260924 00:13:27.775516 20049 catalog_manager.cc:4841] TS 37bd7ff7d5e344f285a9b8648883a87c (127.0.98.3:44729): tablet a5e16cf918b948ad8b8fb22cbadf0377 (table test-table [id=dcf00348977047c093a19a8ee5c3a8c4]) successfully deleted
W20260924 00:13:27.775570 20047 catalog_manager.cc:4584] TS f780c66bd3584c69a4cb96f8b82ec853 (127.0.98.1:41681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5e16cf918b948ad8b8fb22cbadf0377: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41681: connect: Connection refused (error 111)
I20260924 00:13:27.778671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20384
I20260924 00:13:27.783484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20031
2026-09-24T00:13:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (744 ms)
[----------] 6 tests from EncryptionPolicyTest (8446 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:27 dist-test-slave-3q8k krb5kdc[20554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:27 dist-test-slave-3q8k krb5kdc[20554](info): set up 2 sockets
Sep 24 00:13:27 dist-test-slave-3q8k krb5kdc[20554](info): commencing operation
krb5kdc: starting...
W20260924 00:13:28.867364 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:28 dist-test-slave-3q8k krb5kdc[20554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:28.994083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:29.074424 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:29.074688 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:29.074728 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:29.075985 20567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:29.076057 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:29.076081 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:29.076105 20567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:29.076145 20567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:29.077667 20567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:29.077962 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:29.078200 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:29.083207 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:29.083254 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:29.083320 20567 server_base.cc:1034] running on GCE node
W20260924 00:13:29.083190 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:29.083637 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:29.083992 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:29.085155 20567 hybrid_clock.cc:648] HybridClock initialized: now 1790208809085143 us; error 43 us; skew 500 ppm
Sep 24 00:13:29 dist-test-slave-3q8k krb5kdc[20554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:29.087759 20567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:29.088164 20567 webserver.cc:460] Webserver started at http://127.0.0.1:43307/ using document root <none> and password file <none>
I20260924 00:13:29.088342 20567 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:29.088390 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:29.088485 20567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:29.089502 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "ba09bef1b316420ea74000ed4c21d8b2"
format_stamp: "Formatted at 2026-09-24 00:13:29 on dist-test-slave-3q8k"
server_key: "476892a3321bb0b974c6009aef1f5a10"
server_key_iv: "16ef60f9052a175a343333e3708797a5"
server_key_version: "encryptionkey@0"
I20260924 00:13:29.089813 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba09bef1b316420ea74000ed4c21d8b2"
format_stamp: "Formatted at 2026-09-24 00:13:29 on dist-test-slave-3q8k"
server_key: "476892a3321bb0b974c6009aef1f5a10"
server_key_iv: "16ef60f9052a175a343333e3708797a5"
server_key_version: "encryptionkey@0"
I20260924 00:13:29.091090 20567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:29.091943 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:29.092097 20567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:29.092190 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "ba09bef1b316420ea74000ed4c21d8b2"
format_stamp: "Formatted at 2026-09-24 00:13:29 on dist-test-slave-3q8k"
server_key: "476892a3321bb0b974c6009aef1f5a10"
server_key_iv: "16ef60f9052a175a343333e3708797a5"
server_key_version: "encryptionkey@0"
I20260924 00:13:29.092247 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:29.107681 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:29.109687 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:29.109799 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:29.113813 20567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36411
I20260924 00:13:29.113848 20634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36411 every 8 connection(s)
I20260924 00:13:29.114224 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:29.115139 20635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:29.117373 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: Bootstrap starting.
I20260924 00:13:29.117965 20635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:29.118244 20635 log.cc:824] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:29.118640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20567
I20260924 00:13:29.118778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:29.118974 392 external_mini_cluster.cc:1428] Setting key 6d42b88918319a935eec2ab0c535703a
I20260924 00:13:29.119081 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: No bootstrap required, opened a new log
I20260924 00:13:29.120992 20635 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } }
I20260924 00:13:29.121147 20635 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:29.121197 20635 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba09bef1b316420ea74000ed4c21d8b2, State: Initialized, Role: FOLLOWER
I20260924 00:13:29.121325 20635 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } }
I20260924 00:13:29.121416 20635 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:29.121460 20635 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:29.121508 20635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:29.122025 20635 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } }
I20260924 00:13:29.122136 20635 leader_election.cc:302] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba09bef1b316420ea74000ed4c21d8b2; no voters:
I20260924 00:13:29.122305 20635 leader_election.cc:288] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:29.122339 20640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:29.122474 20640 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [term 1 LEADER]: Becoming Leader. State: Replica: ba09bef1b316420ea74000ed4c21d8b2, State: Running, Role: LEADER
I20260924 00:13:29.122570 20635 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:29.122665 20640 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } }
I20260924 00:13:29.122998 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba09bef1b316420ea74000ed4c21d8b2. Latest consensus state: current_term: 1 leader_uuid: "ba09bef1b316420ea74000ed4c21d8b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } } }
I20260924 00:13:29.123076 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:29.123953 20641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba09bef1b316420ea74000ed4c21d8b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba09bef1b316420ea74000ed4c21d8b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36411 } } }
I20260924 00:13:29.124241 20641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2 [sys.catalog]: This master's current role is: LEADER
W20260924 00:13:29.124493 20656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:29.124549 20656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:29.124753 20646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:29.124958 20646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:29.126634 20646 catalog_manager.cc:1347] Generated new cluster ID: 4619ce52b00f4e7f88592a1af4716826
Sep 24 00:13:29 dist-test-slave-3q8k krb5kdc[20554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:13:29.126688 20646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:29.131323 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:29.119547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48767 (local address 127.0.0.1:36411)
0924 00:13:29.119724 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:29.119733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:29.119764 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:13:29.120326 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:29.120335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:29.120380 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:29.120545 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:29.121163 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:29.122580 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:29.123655 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:29.124322 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:29.124428 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:29.127206 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:29.127210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:29.127210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:29.127243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:29.129242 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:29.129671 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:29.129674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:29.129675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:29.129782 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:29.130029 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:29.130030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:29.130030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:29.130331 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:29.130895 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:29.130987 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 00:13:29.131094 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260924 00:13:29.132256 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:29.140792 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:29.133739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39911 (local address 127.0.0.1:36411)
0924 00:13:29.133776 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:29.133778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:29.133934 (+ 156us) server_negotiation.cc:407] Connection header received
0924 00:13:29.133979 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:29.133980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:29.133989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:29.134057 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:29.134616 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:29.135403 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:29.136333 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:29.136669 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:29.136740 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:29.138041 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:29.138042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:29.138043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:29.138049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:29.139600 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:29.139982 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:29.139983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:29.139984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:29.140040 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:29.140256 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:29.140258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:29.140258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:29.140389 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:29.140485 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:29.140630 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:13:29.140698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:13:29.142131 20646 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:29.142621 20646 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:29.150094 20646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba09bef1b316420ea74000ed4c21d8b2: Generated new TSK 0
I20260924 00:13:29.150296 20646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:29.152916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20567
W20260924 00:13:29.157320 20552 connection.cc:535] client connection to 127.0.0.1:36411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36411): Connection reset by peer (error 104)
2026-09-24T00:13:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:29 dist-test-slave-3q8k krb5kdc[20666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:29 dist-test-slave-3q8k krb5kdc[20666](info): set up 2 sockets
Sep 24 00:13:29 dist-test-slave-3q8k krb5kdc[20666](info): commencing operation
krb5kdc: starting...
W20260924 00:13:30.223318 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:30 dist-test-slave-3q8k krb5kdc[20666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:30.344461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:30.426487 20679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:30.426718 20679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:30.426759 20679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:30.427934 20679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:30.427999 20679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:30.428023 20679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:30.428050 20679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:30.428082 20679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:30.429549 20679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:30.429826 20679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:30.430045 20679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:30.434816 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:30.434895 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:30.434825 20687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:30.435068 20679 server_base.cc:1034] running on GCE node
I20260924 00:13:30.435356 20679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:30.435629 20679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:30.436796 20679 hybrid_clock.cc:648] HybridClock initialized: now 1790208810436785 us; error 42 us; skew 500 ppm
Sep 24 00:13:30 dist-test-slave-3q8k krb5kdc[20666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:30.445633 20679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:30.446157 20679 webserver.cc:460] Webserver started at http://127.0.0.1:43107/ using document root <none> and password file <none>
I20260924 00:13:30.446344 20679 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:30.446386 20679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:30.446475 20679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:30.447587 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "8d2c652d2f6342f4a52b3990e404b5d0"
format_stamp: "Formatted at 2026-09-24 00:13:30 on dist-test-slave-3q8k"
server_key: "3a6cd1b43718ca13905e277dda4ee3dd"
server_key_iv: "4698ad79998782416d20dde2ca126d0f"
server_key_version: "encryptionkey@0"
I20260924 00:13:30.447901 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d2c652d2f6342f4a52b3990e404b5d0"
format_stamp: "Formatted at 2026-09-24 00:13:30 on dist-test-slave-3q8k"
server_key: "3a6cd1b43718ca13905e277dda4ee3dd"
server_key_iv: "4698ad79998782416d20dde2ca126d0f"
server_key_version: "encryptionkey@0"
I20260924 00:13:30.449174 20679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:30.449882 20694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:30.450078 20679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:30.450148 20679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8d2c652d2f6342f4a52b3990e404b5d0"
format_stamp: "Formatted at 2026-09-24 00:13:30 on dist-test-slave-3q8k"
server_key: "3a6cd1b43718ca13905e277dda4ee3dd"
server_key_iv: "4698ad79998782416d20dde2ca126d0f"
server_key_version: "encryptionkey@0"
I20260924 00:13:30.450201 20679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:30.493295 20679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:30.495411 20679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:30.495527 20679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:30.499188 20679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45237
I20260924 00:13:30.499249 20746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45237 every 8 connection(s)
I20260924 00:13:30.499686 20679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:30.500578 20747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:30.502898 20747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0: Bootstrap starting.
I20260924 00:13:30.503597 20747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:30.503942 20747 log.cc:824] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:30.504647 20747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0: No bootstrap required, opened a new log
I20260924 00:13:30.506279 20747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260924 00:13:30.506475 20747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:30.506529 20747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2c652d2f6342f4a52b3990e404b5d0, State: Initialized, Role: FOLLOWER
I20260924 00:13:30.506680 20747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260924 00:13:30.506762 20747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:30.506806 20747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:30.506858 20747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:30.507613 20747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260924 00:13:30.507742 20747 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2c652d2f6342f4a52b3990e404b5d0; no voters:
I20260924 00:13:30.507910 20747 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:30.508082 20751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:30.508190 20747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:30.508234 20751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d2c652d2f6342f4a52b3990e404b5d0, State: Running, Role: LEADER
I20260924 00:13:30.508399 20751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260924 00:13:30.509147 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } } }
I20260924 00:13:30.509145 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2c652d2f6342f4a52b3990e404b5d0. Latest consensus state: current_term: 1 leader_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2c652d2f6342f4a52b3990e404b5d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } } }
I20260924 00:13:30.509270 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:30.509274 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:30.509555 20762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:30.509781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20679
I20260924 00:13:30.509902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:30.510099 392 external_mini_cluster.cc:1428] Setting key 1046fb9e1d32e039ba740d57f064c9f7
I20260924 00:13:30.510141 20762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:30.511646 20762 catalog_manager.cc:1347] Generated new cluster ID: e4142cf33b9040b3864116b308255dc2
I20260924 00:13:30.511696 20762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:30.516721 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:30.510534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35463 (local address 127.0.0.1:45237)
0924 00:13:30.510721 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:30.510729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:30.511196 (+ 467us) server_negotiation.cc:407] Connection header received
0924 00:13:30.511375 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:30.511386 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:30.511443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:30.511860 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:30.511957 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:30.513356 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:30.514463 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:30.515127 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:30.515280 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:30.516012 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:30.516016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:30.516017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:30.516228 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:30.516268 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:30.516284 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:30.516376 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:30.516461 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 00:13:30.516490 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 00:13:30.517940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:30.523063 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:30.519568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38249 (local address 127.0.0.1:45237)
0924 00:13:30.519601 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:30.519603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:30.519610 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:30.519694 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:30.519695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:30.519704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:30.519753 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:30.519889 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:30.520607 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:30.521639 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:30.521935 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:30.521987 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:30.522709 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:30.522710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:30.522710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:30.522714 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:30.522728 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:30.522732 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:30.522781 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:30.522886 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:13:30.522982 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:13:30.528509 20762 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:30.529230 20762 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:30.534402 20762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2c652d2f6342f4a52b3990e404b5d0: Generated new TSK 0
I20260924 00:13:30.534631 20762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:30.539696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20679
W20260924 00:13:30.543627 20664 connection.cc:535] client connection to 127.0.0.1:45237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45237): Connection reset by peer (error 104)
2026-09-24T00:13:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:30 dist-test-slave-3q8k krb5kdc[20778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:30 dist-test-slave-3q8k krb5kdc[20778](info): set up 2 sockets
Sep 24 00:13:30 dist-test-slave-3q8k krb5kdc[20778](info): commencing operation
krb5kdc: starting...
W20260924 00:13:31.595285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:31 dist-test-slave-3q8k krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:31.719762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:31.801060 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:31.801287 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:31.801311 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:31.802496 20791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:31.802549 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:31.802564 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:31.802577 20791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:31.802589 20791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:31.804402 20791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:31.804675 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:31.804914 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:31.809806 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:31.809916 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:31.809923 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:31.810102 20791 server_base.cc:1034] running on GCE node
I20260924 00:13:31.810370 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:31.810619 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:31.811803 20791 hybrid_clock.cc:648] HybridClock initialized: now 1790208811811790 us; error 31 us; skew 500 ppm
Sep 24 00:13:31 dist-test-slave-3q8k krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:31.814217 20791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:31.814679 20791 webserver.cc:460] Webserver started at http://127.0.0.1:36627/ using document root <none> and password file <none>
I20260924 00:13:31.814867 20791 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:31.814908 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:31.814998 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:31.816062 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "5821d664d0ae4f8fb854a3e135fa7d19"
format_stamp: "Formatted at 2026-09-24 00:13:31 on dist-test-slave-3q8k"
server_key: "f4822ec5eee8078dcebef92f57cd4a5f"
server_key_iv: "270e200800366f37c2a6829ea3a0d1b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:31.816358 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "5821d664d0ae4f8fb854a3e135fa7d19"
format_stamp: "Formatted at 2026-09-24 00:13:31 on dist-test-slave-3q8k"
server_key: "f4822ec5eee8078dcebef92f57cd4a5f"
server_key_iv: "270e200800366f37c2a6829ea3a0d1b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:31.817618 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:13:31.818378 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:31.818583 20791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:31.818682 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "5821d664d0ae4f8fb854a3e135fa7d19"
format_stamp: "Formatted at 2026-09-24 00:13:31 on dist-test-slave-3q8k"
server_key: "f4822ec5eee8078dcebef92f57cd4a5f"
server_key_iv: "270e200800366f37c2a6829ea3a0d1b4"
server_key_version: "encryptionkey@0"
I20260924 00:13:31.818819 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:31.822141 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:31.824380 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:31.824527 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:31.828577 20791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38077
I20260924 00:13:31.828608 20858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38077 every 8 connection(s)
I20260924 00:13:31.829002 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:31.829977 20859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:31.832415 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19: Bootstrap starting.
I20260924 00:13:31.833065 20859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:31.833366 20859 log.cc:824] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:31.834156 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19: No bootstrap required, opened a new log
I20260924 00:13:31.834316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20791
I20260924 00:13:31.834438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:31.834640 392 external_mini_cluster.cc:1428] Setting key dea804efc4c22da7e494d3057de76075
I20260924 00:13:31.836229 20859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260924 00:13:31.836422 20859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:31.836459 20859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5821d664d0ae4f8fb854a3e135fa7d19, State: Initialized, Role: FOLLOWER
I20260924 00:13:31.836650 20859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260924 00:13:31.836751 20859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:31.836786 20859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:31.836867 20859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:31.837695 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:31.835157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49115 (local address 127.0.0.1:38077)
0924 00:13:31.835358 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:31.835366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:31.835392 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:13:31.836240 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:31.836264 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:31.836297 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:31.836458 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:31.837157 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:31.837161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:31.837162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:31.837167 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:31.837225 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:31.837241 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:31.837324 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:31.837382 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 00:13:31.837430 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260924 00:13:31.838106 20859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260924 00:13:31.838218 20859 leader_election.cc:302] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5821d664d0ae4f8fb854a3e135fa7d19; no voters:
I20260924 00:13:31.838546 20864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:31.838585 20859 leader_election.cc:288] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:31.838773 20864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [term 1 LEADER]: Becoming Leader. State: Replica: 5821d664d0ae4f8fb854a3e135fa7d19, State: Running, Role: LEADER
I20260924 00:13:31.838899 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:31.838928 20864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260924 00:13:31.838990 20859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:31.839291 20865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5821d664d0ae4f8fb854a3e135fa7d19. Latest consensus state: current_term: 1 leader_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260924 00:13:31.839362 20865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:31.839491 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5821d664d0ae4f8fb854a3e135fa7d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260924 00:13:31.839553 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:31.839944 20868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:31.840440 20868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:31.841926 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:31.840825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41371 (local address 127.0.0.1:38077)
0924 00:13:31.840864 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:31.840866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:31.840970 (+ 104us) server_negotiation.cc:407] Connection header received
0924 00:13:31.841017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:31.841018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:31.841025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:31.841084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:31.841692 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:31.841693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:31.841693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:31.841698 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:31.841711 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:31.841715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:31.841773 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:31.841827 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:13:31.841841 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:31.842082 20868 catalog_manager.cc:1347] Generated new cluster ID: 86ac00edbebd41468b9c6ec6de3b47ba
I20260924 00:13:31.842183 20868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:31.854233 20868 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:31.854417 20868 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:31.860741 20868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5821d664d0ae4f8fb854a3e135fa7d19: Generated new TSK 0
I20260924 00:13:31.860939 20868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:31.867563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20791
2026-09-24T00:13:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:31 dist-test-slave-3q8k krb5kdc[20890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:31 dist-test-slave-3q8k krb5kdc[20890](info): set up 2 sockets
Sep 24 00:13:31 dist-test-slave-3q8k krb5kdc[20890](info): commencing operation
krb5kdc: starting...
W20260924 00:13:32.895383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:32 dist-test-slave-3q8k krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:33.022773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:33.103832 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:33.104082 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:33.104120 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:33.105345 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:33.105419 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:33.105443 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:33.105468 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:33.105494 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:33.107050 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:33.107378 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:33.107610 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:33.112432 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:33.112473 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:33.112465 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:33.112612 20903 server_base.cc:1034] running on GCE node
I20260924 00:13:33.112936 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:33.113245 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:33.114415 20903 hybrid_clock.cc:648] HybridClock initialized: now 1790208813114403 us; error 27 us; skew 500 ppm
Sep 24 00:13:33 dist-test-slave-3q8k krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:33.117316 20903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:33.117789 20903 webserver.cc:460] Webserver started at http://127.0.0.1:39017/ using document root <none> and password file <none>
I20260924 00:13:33.118099 20903 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:33.118268 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:33.118431 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:33.119621 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "bd73b89ed65b4ac2baed2b7ad66845ec"
format_stamp: "Formatted at 2026-09-24 00:13:33 on dist-test-slave-3q8k"
server_key: "ecdd5757b3b09c91d5e0b1cbac7fdc49"
server_key_iv: "660930472ba2334a1e1b20f80da220cb"
server_key_version: "encryptionkey@0"
I20260924 00:13:33.119995 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd73b89ed65b4ac2baed2b7ad66845ec"
format_stamp: "Formatted at 2026-09-24 00:13:33 on dist-test-slave-3q8k"
server_key: "ecdd5757b3b09c91d5e0b1cbac7fdc49"
server_key_iv: "660930472ba2334a1e1b20f80da220cb"
server_key_version: "encryptionkey@0"
I20260924 00:13:33.121384 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:33.122226 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:33.122416 20903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:33.122519 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "bd73b89ed65b4ac2baed2b7ad66845ec"
format_stamp: "Formatted at 2026-09-24 00:13:33 on dist-test-slave-3q8k"
server_key: "ecdd5757b3b09c91d5e0b1cbac7fdc49"
server_key_iv: "660930472ba2334a1e1b20f80da220cb"
server_key_version: "encryptionkey@0"
I20260924 00:13:33.122592 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:33.143177 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:33.145252 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:33.145416 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:33.149227 20903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35829
I20260924 00:13:33.149276 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35829 every 8 connection(s)
I20260924 00:13:33.149641 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:33.150615 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:33.153051 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec: Bootstrap starting.
I20260924 00:13:33.153693 20971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:33.153988 20971 log.cc:824] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:33.154740 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec: No bootstrap required, opened a new log
I20260924 00:13:33.156389 20971 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260924 00:13:33.156529 20971 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:33.156574 20971 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd73b89ed65b4ac2baed2b7ad66845ec, State: Initialized, Role: FOLLOWER
I20260924 00:13:33.156744 20971 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260924 00:13:33.156826 20971 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:33.156874 20971 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:33.156926 20971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:33.157485 20971 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260924 00:13:33.157595 20971 leader_election.cc:302] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd73b89ed65b4ac2baed2b7ad66845ec; no voters:
I20260924 00:13:33.157778 20971 leader_election.cc:288] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:33.157833 20974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:33.157838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 20903
I20260924 00:13:33.157961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:33.157995 20974 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [term 1 LEADER]: Becoming Leader. State: Replica: bd73b89ed65b4ac2baed2b7ad66845ec, State: Running, Role: LEADER
I20260924 00:13:33.158061 20971 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:33.158084 20974 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } }
I20260924 00:13:33.158156 392 external_mini_cluster.cc:1428] Setting key c6f77d7d999ab6bbffca9be18655f663
I20260924 00:13:33.158434 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd73b89ed65b4ac2baed2b7ad66845ec. Latest consensus state: current_term: 1 leader_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260924 00:13:33.158568 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd73b89ed65b4ac2baed2b7ad66845ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35829 } } }
I20260924 00:13:33.158568 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:33.158653 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:33.158854 20984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:33.159435 20984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:33.161191 20984 catalog_manager.cc:1347] Generated new cluster ID: 5e1b66a6ef544dd2a68b432a65ee9b1c
I20260924 00:13:33.161249 20984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:13:33 dist-test-slave-3q8k krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:13:33.170912 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:33.158846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56719 (local address 127.0.0.1:35829)
0924 00:13:33.159001 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:33.159011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:33.159228 (+ 217us) server_negotiation.cc:407] Connection header received
0924 00:13:33.159460 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:33.159468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:33.159530 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:33.159686 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:33.160711 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:33.162062 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:33.163190 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:33.163873 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:33.164312 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:33.166869 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:33.166874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:33.166874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:33.166889 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:33.169066 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:33.169477 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:33.169481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:33.169481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:33.169623 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:33.169835 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:33.169836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:33.169836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:33.170238 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:33.170407 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:33.170512 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:13:33.170662 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260924 00:13:33.171950 20984 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:33.172377 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:33.172614 20984 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:33.178689 20984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd73b89ed65b4ac2baed2b7ad66845ec: Generated new TSK 0
I20260924 00:13:33.178910 20984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:33.181562 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:33.173877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56913 (local address 127.0.0.1:35829)
0924 00:13:33.173919 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:33.173921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:33.174025 (+ 104us) server_negotiation.cc:407] Connection header received
0924 00:13:33.174084 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:33.174085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:33.174096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:33.174165 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:33.174786 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:33.175637 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:33.176678 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:33.177015 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:33.177085 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:33.178717 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:33.178718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:33.178719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:33.178726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:33.180478 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:33.180837 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:33.180838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:33.180839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:33.180905 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:33.181076 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:33.181077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:33.181077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:33.181207 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:33.181304 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:33.181387 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 00:13:33.181469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:33.183642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 20903
2026-09-24T00:13:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:33 dist-test-slave-3q8k krb5kdc[21002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:33 dist-test-slave-3q8k krb5kdc[21002](info): set up 2 sockets
Sep 24 00:13:33 dist-test-slave-3q8k krb5kdc[21002](info): commencing operation
krb5kdc: starting...
W20260924 00:13:34.231281 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:34 dist-test-slave-3q8k krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:34.350394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:34.430668 21015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:34.430876 21015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:34.430898 21015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:34.432062 21015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:34.432108 21015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:34.432122 21015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:34.432173 21015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:34.432199 21015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:34.433638 21015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:34.433883 21015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:34.434099 21015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:34.438967 21020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:34.439000 21023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:34.439271 21021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:34.439406 21015 server_base.cc:1034] running on GCE node
I20260924 00:13:34.439647 21015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:34.440006 21015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:34.441143 21015 hybrid_clock.cc:648] HybridClock initialized: now 1790208814441121 us; error 36 us; skew 500 ppm
Sep 24 00:13:34 dist-test-slave-3q8k krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:34.443775 21015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:34.444145 21015 webserver.cc:460] Webserver started at http://127.0.0.1:34787/ using document root <none> and password file <none>
I20260924 00:13:34.444322 21015 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:34.444362 21015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:34.444451 21015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:34.445447 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "ef0a263a3d6443d088c94fbe6ffe134d"
format_stamp: "Formatted at 2026-09-24 00:13:34 on dist-test-slave-3q8k"
server_key: "0f9f36038157f1091cd1b5fc9154e845"
server_key_iv: "0b079e3a8bf825af62babfa8542d4814"
server_key_version: "encryptionkey@0"
I20260924 00:13:34.445741 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef0a263a3d6443d088c94fbe6ffe134d"
format_stamp: "Formatted at 2026-09-24 00:13:34 on dist-test-slave-3q8k"
server_key: "0f9f36038157f1091cd1b5fc9154e845"
server_key_iv: "0b079e3a8bf825af62babfa8542d4814"
server_key_version: "encryptionkey@0"
I20260924 00:13:34.446978 21015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:34.447677 21030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:34.447866 21015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:34.447963 21015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "ef0a263a3d6443d088c94fbe6ffe134d"
format_stamp: "Formatted at 2026-09-24 00:13:34 on dist-test-slave-3q8k"
server_key: "0f9f36038157f1091cd1b5fc9154e845"
server_key_iv: "0b079e3a8bf825af62babfa8542d4814"
server_key_version: "encryptionkey@0"
I20260924 00:13:34.448040 21015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:34.460426 21015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:34.462731 21015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:34.462898 21015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:34.466825 21015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46653
I20260924 00:13:34.466869 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46653 every 8 connection(s)
I20260924 00:13:34.467298 21015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:34.468142 21083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:34.470333 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d: Bootstrap starting.
I20260924 00:13:34.470896 21083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:34.471171 21083 log.cc:824] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:34.471907 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d: No bootstrap required, opened a new log
I20260924 00:13:34.473434 21083 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260924 00:13:34.473574 21083 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:34.473598 21083 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a263a3d6443d088c94fbe6ffe134d, State: Initialized, Role: FOLLOWER
I20260924 00:13:34.473786 21083 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260924 00:13:34.473905 21083 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:34.473928 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:34.473977 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:34.474529 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260924 00:13:34.474614 21083 leader_election.cc:302] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef0a263a3d6443d088c94fbe6ffe134d; no voters:
I20260924 00:13:34.474839 21083 leader_election.cc:288] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:34.474901 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:34.475046 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a263a3d6443d088c94fbe6ffe134d, State: Running, Role: LEADER
I20260924 00:13:34.475113 21083 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:34.475142 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260924 00:13:34.475183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21015
I20260924 00:13:34.475344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:34.475534 392 external_mini_cluster.cc:1428] Setting key 25b51c29ab7ddb2336fb9fd6bb7ec26f
I20260924 00:13:34.475559 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0a263a3d6443d088c94fbe6ffe134d. Latest consensus state: current_term: 1 leader_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } } }
I20260924 00:13:34.475633 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:34.475693 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0a263a3d6443d088c94fbe6ffe134d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } } }
I20260924 00:13:34.475749 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:34.475878 21093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:34.476590 21093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:34.478307 21093 catalog_manager.cc:1347] Generated new cluster ID: 393e99502d4a44b4976802563ad545e4
I20260924 00:13:34.478360 21093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:34.481402 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:34.476161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46955 (local address 127.0.0.1:46653)
0924 00:13:34.476761 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:34.476772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:34.476811 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:13:34.477000 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:34.477007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:34.477052 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:34.477188 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:34.477397 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:34.478633 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:34.479510 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:34.479941 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:34.480063 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:34.480748 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:34.480752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:34.480752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:34.480926 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:34.480955 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:34.480967 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:34.481042 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:34.481114 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:13:34.481185 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":89,"threads_started":1}
I20260924 00:13:34.482262 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:34.486577 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:34.483665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58639 (local address 127.0.0.1:46653)
0924 00:13:34.483701 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:34.483704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:34.483712 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:34.483778 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:34.483779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:34.483789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:34.483843 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:34.483959 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:34.484428 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:34.485126 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:34.485345 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:34.485414 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:34.486275 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:34.486276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:34.486277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:34.486283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:34.486298 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:34.486304 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:34.486337 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:34.486455 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:13:34.486486 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:13:34.487548 21093 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:34.488263 21093 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:34.500658 21093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef0a263a3d6443d088c94fbe6ffe134d: Generated new TSK 0
I20260924 00:13:34.500886 21093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:34.503458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21015
2026-09-24T00:13:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:34 dist-test-slave-3q8k krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:34 dist-test-slave-3q8k krb5kdc[21114](info): set up 2 sockets
Sep 24 00:13:34 dist-test-slave-3q8k krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260924 00:13:35.527144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:35 dist-test-slave-3q8k krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:35.643101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:35.725399 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:35.725641 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:35.725682 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:35.726846 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:35.726909 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:35.726930 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:35.726953 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:35.726979 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:35.728477 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:35.728765 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:35.728986 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:35.733767 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:35.733778 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:35.733774 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:35.733958 21127 server_base.cc:1034] running on GCE node
I20260924 00:13:35.734203 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:35.734496 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:35.735653 21127 hybrid_clock.cc:648] HybridClock initialized: now 1790208815735628 us; error 54 us; skew 500 ppm
Sep 24 00:13:35 dist-test-slave-3q8k krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:35.744405 21127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:35.744876 21127 webserver.cc:460] Webserver started at http://127.0.0.1:40149/ using document root <none> and password file <none>
I20260924 00:13:35.745106 21127 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:35.745173 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:35.745323 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:35.746420 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3"
format_stamp: "Formatted at 2026-09-24 00:13:35 on dist-test-slave-3q8k"
server_key: "a6e95a9a9210663b0119cd3281a3ab21"
server_key_iv: "144301183faa7b9daffd115a44d4cbe7"
server_key_version: "encryptionkey@0"
I20260924 00:13:35.746783 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3"
format_stamp: "Formatted at 2026-09-24 00:13:35 on dist-test-slave-3q8k"
server_key: "a6e95a9a9210663b0119cd3281a3ab21"
server_key_iv: "144301183faa7b9daffd115a44d4cbe7"
server_key_version: "encryptionkey@0"
I20260924 00:13:35.748186 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:35.748986 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:35.749148 21127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:13:35.749246 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3"
format_stamp: "Formatted at 2026-09-24 00:13:35 on dist-test-slave-3q8k"
server_key: "a6e95a9a9210663b0119cd3281a3ab21"
server_key_iv: "144301183faa7b9daffd115a44d4cbe7"
server_key_version: "encryptionkey@0"
I20260924 00:13:35.749325 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:35.752607 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:35.754369 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:35.754518 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:35.758380 21127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260924 00:13:35.758412 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260924 00:13:35.758750 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:35.759783 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:35.761976 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3: Bootstrap starting.
I20260924 00:13:35.762542 21195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:35.762815 21195 log.cc:824] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:35.763512 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3: No bootstrap required, opened a new log
I20260924 00:13:35.765072 21195 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260924 00:13:35.765230 21195 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:35.765255 21195 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8563aeb97e4bdc85a016b94cbbf1c3, State: Initialized, Role: FOLLOWER
I20260924 00:13:35.765434 21195 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260924 00:13:35.765512 21195 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:35.765556 21195 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:35.765606 21195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:35.766103 21195 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260924 00:13:35.766204 21195 leader_election.cc:302] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8563aeb97e4bdc85a016b94cbbf1c3; no voters:
I20260924 00:13:35.766382 21195 leader_election.cc:288] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:35.766479 21198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:35.766626 21195 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:35.766631 21198 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [term 1 LEADER]: Becoming Leader. State: Replica: dd8563aeb97e4bdc85a016b94cbbf1c3, State: Running, Role: LEADER
I20260924 00:13:35.766768 21198 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260924 00:13:35.767328 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260924 00:13:35.767387 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd8563aeb97e4bdc85a016b94cbbf1c3. Latest consensus state: current_term: 1 leader_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8563aeb97e4bdc85a016b94cbbf1c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260924 00:13:35.767413 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:35.767447 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:35.767684 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:35.767740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21127
I20260924 00:13:35.767860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:35.768049 392 external_mini_cluster.cc:1428] Setting key 8cc370b0b83a4c112b33e718ab89810b
I20260924 00:13:35.768224 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:35.769937 21207 catalog_manager.cc:1347] Generated new cluster ID: 769b6f1156c54c4ca888c38aa7150c0c
I20260924 00:13:35.769992 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:35.770840 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:35.768638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42385 (local address 127.0.0.1:40497)
0924 00:13:35.768995 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:35.769004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:35.769189 (+ 185us) server_negotiation.cc:407] Connection header received
0924 00:13:35.769471 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:35.769477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:35.769501 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:35.769648 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:35.770331 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:35.770334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:35.770334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:35.770338 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:35.770377 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:35.770390 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:35.770451 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:35.770538 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 00:13:35.770665 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":74,"threads_started":1}
I20260924 00:13:35.771727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:35.774241 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:35.773152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41297 (local address 127.0.0.1:40497)
0924 00:13:35.773188 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:35.773190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:35.773378 (+ 188us) server_negotiation.cc:407] Connection header received
0924 00:13:35.773439 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:35.773439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:35.773444 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:35.773494 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:35.774056 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:35.774057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:35.774057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:35.774060 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:35.774071 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:35.774074 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:35.774111 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:35.774172 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 00:13:35.774184 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:35.781819 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:35.781986 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:35.796819 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd8563aeb97e4bdc85a016b94cbbf1c3: Generated new TSK 0
I20260924 00:13:35.797035 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:35.800479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21127
2026-09-24T00:13:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:35 dist-test-slave-3q8k krb5kdc[21226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:35 dist-test-slave-3q8k krb5kdc[21226](info): set up 2 sockets
Sep 24 00:13:35 dist-test-slave-3q8k krb5kdc[21226](info): commencing operation
krb5kdc: starting...
W20260924 00:13:36.831359 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:36 dist-test-slave-3q8k krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:36.953960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:37.034457 21239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:37.034691 21239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:37.034716 21239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:37.035887 21239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:37.035926 21239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:37.035940 21239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:37.035954 21239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:37.035966 21239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:37.037446 21239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:37.037760 21239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:37.038008 21239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:37.043018 21245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:37.043028 21244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:37.043129 21239 server_base.cc:1034] running on GCE node
W20260924 00:13:37.043041 21247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:37.043473 21239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:37.043771 21239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:37.044934 21239 hybrid_clock.cc:648] HybridClock initialized: now 1790208817044933 us; error 40 us; skew 500 ppm
Sep 24 00:13:37 dist-test-slave-3q8k krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:37.047590 21239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:37.048090 21239 webserver.cc:460] Webserver started at http://127.0.0.1:40157/ using document root <none> and password file <none>
I20260924 00:13:37.048321 21239 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:37.048492 21239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:37.048666 21239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:37.049850 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "978a5e5e22894fe4bbc38917de1ab1bf"
format_stamp: "Formatted at 2026-09-24 00:13:37 on dist-test-slave-3q8k"
server_key: "dc71acd87f30739fdedb65715466afd7"
server_key_iv: "c5f3e9f5a07d4657e7091e80ab86b72b"
server_key_version: "encryptionkey@0"
I20260924 00:13:37.050225 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "978a5e5e22894fe4bbc38917de1ab1bf"
format_stamp: "Formatted at 2026-09-24 00:13:37 on dist-test-slave-3q8k"
server_key: "dc71acd87f30739fdedb65715466afd7"
server_key_iv: "c5f3e9f5a07d4657e7091e80ab86b72b"
server_key_version: "encryptionkey@0"
I20260924 00:13:37.051657 21239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:37.052426 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:37.052620 21239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:37.052735 21239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "978a5e5e22894fe4bbc38917de1ab1bf"
format_stamp: "Formatted at 2026-09-24 00:13:37 on dist-test-slave-3q8k"
server_key: "dc71acd87f30739fdedb65715466afd7"
server_key_iv: "c5f3e9f5a07d4657e7091e80ab86b72b"
server_key_version: "encryptionkey@0"
I20260924 00:13:37.052810 21239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:37.061422 21239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:37.063747 21239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:37.063911 21239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:37.068017 21239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43547
I20260924 00:13:37.068063 21306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43547 every 8 connection(s)
I20260924 00:13:37.068442 21239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:37.068763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21239
I20260924 00:13:37.068897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:37.069094 392 external_mini_cluster.cc:1428] Setting key f65b86f2551a59b5f4f14f5b7e4c85fd
I20260924 00:13:37.069448 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:37.073071 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf: Bootstrap starting.
I20260924 00:13:37.073696 21307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:37.074039 21307 log.cc:824] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:37.074852 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf: No bootstrap required, opened a new log
Sep 24 00:13:37 dist-test-slave-3q8k krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:13:37.077014 21307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260924 00:13:37.077211 21307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:37.077253 21307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978a5e5e22894fe4bbc38917de1ab1bf, State: Initialized, Role: FOLLOWER
I20260924 00:13:37.077417 21307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260924 00:13:37.077510 21307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:37.077553 21307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:37.077613 21307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:37.078171 21307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260924 00:13:37.078290 21307 leader_election.cc:302] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978a5e5e22894fe4bbc38917de1ab1bf; no voters:
I20260924 00:13:37.078473 21307 leader_election.cc:288] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:37.078533 21312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:37.078773 21307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:37.078836 21312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [term 1 LEADER]: Becoming Leader. State: Replica: 978a5e5e22894fe4bbc38917de1ab1bf, State: Running, Role: LEADER
I20260924 00:13:37.078970 21312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260924 00:13:37.079471 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978a5e5e22894fe4bbc38917de1ab1bf. Latest consensus state: current_term: 1 leader_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260924 00:13:37.079586 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:37.079460 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a5e5e22894fe4bbc38917de1ab1bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260924 00:13:37.079722 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:37.080149 21322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:37.081043 21322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:37.081737 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:37.069615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:41515 (local address 127.0.0.1:43547)
0924 00:13:37.069863 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:37.069873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:37.069905 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:13:37.070584 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:37.070594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:37.070663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:37.070827 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:37.071562 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:37.072991 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:37.073864 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:37.074444 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:37.074584 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:37.077058 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:37.077062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:37.077062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:37.077083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:37.079409 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:37.079931 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:37.079934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:37.079935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:37.080042 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:37.080364 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:37.080365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:37.080366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:37.080709 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:37.081332 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:37.081430 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 00:13:37.081528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":140,"threads_started":1}
I20260924 00:13:37.082679 21322 catalog_manager.cc:1347] Generated new cluster ID: 9ed40b1c60384ec28e9d0f653ef8029a
I20260924 00:13:37.082734 21322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:37.082903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:37.092622 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:37.084484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:33611 (local address 127.0.0.1:43547)
0924 00:13:37.084523 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:37.084525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:37.084643 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:13:37.084738 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:37.084739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:37.084749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:37.084778 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:37.085515 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:37.086297 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:37.087470 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:37.087788 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:37.087833 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:37.089478 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:37.089479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:37.089479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:37.089486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:37.091195 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:37.091679 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:37.091680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:37.091681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:37.091737 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:37.091989 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:37.091990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:37.091990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:37.092137 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:37.092217 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:37.092458 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 00:13:37.092523 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:37.095705 21322 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:37.096419 21322 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:37.104025 21322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978a5e5e22894fe4bbc38917de1ab1bf: Generated new TSK 0
I20260924 00:13:37.104254 21322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:37.111549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21239
W20260924 00:13:37.115510 21224 connection.cc:535] client connection to 127.0.0.1:43547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43547): Connection reset by peer (error 104)
2026-09-24T00:13:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:37 dist-test-slave-3q8k krb5kdc[21338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:37 dist-test-slave-3q8k krb5kdc[21338](info): set up 2 sockets
Sep 24 00:13:37 dist-test-slave-3q8k krb5kdc[21338](info): commencing operation
krb5kdc: starting...
W20260924 00:13:38.151436 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:38 dist-test-slave-3q8k krb5kdc[21338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:38.274778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:38.355700 21351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:38.355921 21351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:38.355944 21351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:38.357079 21351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:38.357121 21351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:38.357136 21351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:38.357148 21351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:38.357160 21351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:38.358618 21351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:38.358827 21351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:38.359014 21351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:38.363984 21359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:38.364017 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:38.364012 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:38.364212 21351 server_base.cc:1034] running on GCE node
I20260924 00:13:38.364420 21351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:38.364689 21351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:38.365980 21351 hybrid_clock.cc:648] HybridClock initialized: now 1790208818365966 us; error 30 us; skew 500 ppm
Sep 24 00:13:38 dist-test-slave-3q8k krb5kdc[21338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:38.368924 21351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:38.369365 21351 webserver.cc:460] Webserver started at http://127.0.0.1:38767/ using document root <none> and password file <none>
I20260924 00:13:38.369596 21351 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:38.369663 21351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:38.369808 21351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:38.370877 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "3db92c2c62584b17bb9ffd36d3e99c11"
format_stamp: "Formatted at 2026-09-24 00:13:38 on dist-test-slave-3q8k"
server_key: "916da7b1af159025f1d5d10a1fcc3082"
server_key_iv: "34ed8671640e438c234cdd6f4351ddd3"
server_key_version: "encryptionkey@0"
I20260924 00:13:38.371337 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "3db92c2c62584b17bb9ffd36d3e99c11"
format_stamp: "Formatted at 2026-09-24 00:13:38 on dist-test-slave-3q8k"
server_key: "916da7b1af159025f1d5d10a1fcc3082"
server_key_iv: "34ed8671640e438c234cdd6f4351ddd3"
server_key_version: "encryptionkey@0"
I20260924 00:13:38.372776 21351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:38.373607 21366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:38.373844 21351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:38.373950 21351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "3db92c2c62584b17bb9ffd36d3e99c11"
format_stamp: "Formatted at 2026-09-24 00:13:38 on dist-test-slave-3q8k"
server_key: "916da7b1af159025f1d5d10a1fcc3082"
server_key_iv: "34ed8671640e438c234cdd6f4351ddd3"
server_key_version: "encryptionkey@0"
I20260924 00:13:38.374034 21351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:38.390762 21351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:38.393272 21351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:38.393440 21351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:38.397415 21351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39325
I20260924 00:13:38.397464 21418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39325 every 8 connection(s)
I20260924 00:13:38.397854 21351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:38.398775 21419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:38.399840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21351
I20260924 00:13:38.399964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:38.400169 392 external_mini_cluster.cc:1428] Setting key bb478d9b853fba0fdbfffb2035e61aa8
I20260924 00:13:38.401616 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11: Bootstrap starting.
I20260924 00:13:38.402263 21419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:38.402544 21419 log.cc:824] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:38.403324 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11: No bootstrap required, opened a new log
I20260924 00:13:38.405866 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:38.400752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:36717 (local address 127.0.0.1:39325)
0924 00:13:38.400914 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:38.400923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:38.400955 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:13:38.401420 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:38.401426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:38.401476 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:38.401599 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:38.401837 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:38.402818 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:38.403701 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:38.404318 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:38.404433 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:38.405119 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:38.405125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:38.405125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:38.405382 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:38.405428 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:38.405446 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:38.405551 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:38.405619 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:13:38.405648 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 00:13:38.405838 21419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } }
I20260924 00:13:38.406008 21419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:38.406068 21419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db92c2c62584b17bb9ffd36d3e99c11, State: Initialized, Role: FOLLOWER
I20260924 00:13:38.406210 21419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } }
I20260924 00:13:38.406289 21419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:38.406320 21419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:38.406368 21419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:38.406877 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:38.407121 21419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } }
I20260924 00:13:38.407258 21419 leader_election.cc:302] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db92c2c62584b17bb9ffd36d3e99c11; no voters:
I20260924 00:13:38.407452 21419 leader_election.cc:288] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:38.407522 21424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:38.407727 21424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [term 1 LEADER]: Becoming Leader. State: Replica: 3db92c2c62584b17bb9ffd36d3e99c11, State: Running, Role: LEADER
I20260924 00:13:38.407737 21419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:38.407830 21424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } }
I20260924 00:13:38.408264 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db92c2c62584b17bb9ffd36d3e99c11. Latest consensus state: current_term: 1 leader_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } } }
I20260924 00:13:38.408279 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db92c2c62584b17bb9ffd36d3e99c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39325 } } }
I20260924 00:13:38.408425 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:38.408736 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:38.409286 21438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:38.409796 21438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:38.411482 21438 catalog_manager.cc:1347] Generated new cluster ID: 8adc9356e4854e37ac4061e52bd6c9db
I20260924 00:13:38.411536 21438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:38.411816 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:38.408612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:35757 (local address 127.0.0.1:39325)
0924 00:13:38.408650 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:38.408652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:38.408700 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:13:38.408816 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:38.408817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:38.408830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:38.408864 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:38.409098 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:38.409722 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:38.410449 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:38.410673 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:38.410758 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:38.411508 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:38.411510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:38.411510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:38.411514 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:38.411530 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:38.411534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:38.411596 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:38.411739 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 00:13:38.411750 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:38.416113 21438 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:38.416843 21438 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:38.421341 21438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db92c2c62584b17bb9ffd36d3e99c11: Generated new TSK 0
I20260924 00:13:38.421525 21438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:38.423877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21351
W20260924 00:13:38.428236 21336 connection.cc:535] client connection to 127.0.0.1:39325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39325): Connection reset by peer (error 104)
2026-09-24T00:13:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:38 dist-test-slave-3q8k krb5kdc[21450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:38 dist-test-slave-3q8k krb5kdc[21450](info): set up 2 sockets
Sep 24 00:13:38 dist-test-slave-3q8k krb5kdc[21450](info): commencing operation
krb5kdc: starting...
W20260924 00:13:39.479521 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:39 dist-test-slave-3q8k krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:39.598815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:39.681635 21463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:39.681902 21463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:39.681944 21463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:39.683128 21463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:39.683213 21463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:39.683277 21463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:39.683311 21463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:39.683332 21463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:39.684855 21463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:39.685153 21463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:39.685406 21463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:39.690388 21471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:39.690383 21468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:39.690440 21469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:39.690719 21463 server_base.cc:1034] running on GCE node
I20260924 00:13:39.690938 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:39.691269 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:39.692436 21463 hybrid_clock.cc:648] HybridClock initialized: now 1790208819692414 us; error 39 us; skew 500 ppm
Sep 24 00:13:39 dist-test-slave-3q8k krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:39.695024 21463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:39.695487 21463 webserver.cc:460] Webserver started at http://127.0.0.1:43649/ using document root <none> and password file <none>
I20260924 00:13:39.695688 21463 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:39.695729 21463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:39.695816 21463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:39.696831 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e"
format_stamp: "Formatted at 2026-09-24 00:13:39 on dist-test-slave-3q8k"
server_key: "27939b3ce476506f52232e757a17e345"
server_key_iv: "0227bacdb1bdd8d350a24828fd644716"
server_key_version: "encryptionkey@0"
I20260924 00:13:39.697134 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e"
format_stamp: "Formatted at 2026-09-24 00:13:39 on dist-test-slave-3q8k"
server_key: "27939b3ce476506f52232e757a17e345"
server_key_iv: "0227bacdb1bdd8d350a24828fd644716"
server_key_version: "encryptionkey@0"
I20260924 00:13:39.698369 21463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:39.699025 21478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:39.699204 21463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:39.699345 21463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e"
format_stamp: "Formatted at 2026-09-24 00:13:39 on dist-test-slave-3q8k"
server_key: "27939b3ce476506f52232e757a17e345"
server_key_iv: "0227bacdb1bdd8d350a24828fd644716"
server_key_version: "encryptionkey@0"
I20260924 00:13:39.699420 21463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:39.702643 21463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:39.704762 21463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:39.704921 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:39.708730 21463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35209
I20260924 00:13:39.708783 21530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35209 every 8 connection(s)
I20260924 00:13:39.709162 21463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:39.710189 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:39.712543 21531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e: Bootstrap starting.
I20260924 00:13:39.713136 21531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:39.713416 21531 log.cc:824] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:39.713832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21463
I20260924 00:13:39.713974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:39.714175 392 external_mini_cluster.cc:1428] Setting key 0db9b116ce5c7a457809045f503dc96f
I20260924 00:13:39.714231 21531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e: No bootstrap required, opened a new log
I20260924 00:13:39.716101 21531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260924 00:13:39.716270 21531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:39.716334 21531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75a5f92f1ed448aa2ab6124d94b4f5e, State: Initialized, Role: FOLLOWER
I20260924 00:13:39.716527 21531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260924 00:13:39.716641 21531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:39.716673 21531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:39.716727 21531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:39.717463 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:39.714687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:33807 (local address 127.0.0.1:35209)
0924 00:13:39.714864 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:39.714874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:39.714910 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:13:39.716098 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:39.716105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:39.716135 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:39.716291 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:39.717012 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:39.717016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:39.717016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:39.717021 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:39.717078 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:39.717090 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:39.717167 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:39.717234 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 00:13:39.717280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260924 00:13:39.717834 21531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260924 00:13:39.717959 21531 leader_election.cc:302] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75a5f92f1ed448aa2ab6124d94b4f5e; no voters:
I20260924 00:13:39.718154 21531 leader_election.cc:288] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:39.718181 21536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:39.718334 21536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [term 1 LEADER]: Becoming Leader. State: Replica: f75a5f92f1ed448aa2ab6124d94b4f5e, State: Running, Role: LEADER
I20260924 00:13:39.718402 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:39.718457 21531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:39.718464 21536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } }
I20260924 00:13:39.718809 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75a5f92f1ed448aa2ab6124d94b4f5e. Latest consensus state: current_term: 1 leader_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } } }
I20260924 00:13:39.718935 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:39.719097 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75a5f92f1ed448aa2ab6124d94b4f5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35209 } } }
I20260924 00:13:39.719161 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:39.719650 21542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:39.720170 21542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:39.721853 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:39.720038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:54893 (local address 127.0.0.1:35209)
0924 00:13:39.720075 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:39.720077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:39.720868 (+ 791us) server_negotiation.cc:407] Connection header received
0924 00:13:39.720925 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:39.720926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:39.720931 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:39.721004 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:39.721612 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:39.721613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:39.721614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:39.721618 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:39.721631 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:39.721635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:39.721695 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:39.721751 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 00:13:39.721765 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:39.729477 21542 catalog_manager.cc:1347] Generated new cluster ID: c1825dea4a64491eb6cb24dd100eed02
I20260924 00:13:39.729552 21542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:39.747815 21542 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:39.748015 21542 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:39.751392 21542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75a5f92f1ed448aa2ab6124d94b4f5e: Generated new TSK 0
I20260924 00:13:39.751652 21542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:39.753472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21463
2026-09-24T00:13:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:39 dist-test-slave-3q8k krb5kdc[21562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:39 dist-test-slave-3q8k krb5kdc[21562](info): set up 2 sockets
Sep 24 00:13:39 dist-test-slave-3q8k krb5kdc[21562](info): commencing operation
krb5kdc: starting...
W20260924 00:13:40.791486 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:40 dist-test-slave-3q8k krb5kdc[21562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:40.912510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:40.992765 21575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:40.993012 21575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:40.993052 21575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:40.994174 21575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:40.994241 21575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:40.994266 21575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:40.994287 21575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:40.994314 21575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:40.995833 21575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:40.996112 21575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:40.996336 21575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:41.001133 21581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:41.001140 21580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:41.001312 21583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:41.001515 21575 server_base.cc:1034] running on GCE node
I20260924 00:13:41.001786 21575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:41.002107 21575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:41.003286 21575 hybrid_clock.cc:648] HybridClock initialized: now 1790208821003272 us; error 34 us; skew 500 ppm
Sep 24 00:13:41 dist-test-slave-3q8k krb5kdc[21562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:41.015913 21575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:41.016449 21575 webserver.cc:460] Webserver started at http://127.0.0.1:38845/ using document root <none> and password file <none>
I20260924 00:13:41.016701 21575 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:41.016770 21575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:41.016916 21575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:41.018051 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "40bdd266bc75479992e1d50ccf744bf6"
format_stamp: "Formatted at 2026-09-24 00:13:41 on dist-test-slave-3q8k"
server_key: "b6787ea39999872381a78fb3a45695c8"
server_key_iv: "8fffbd035729457ae97b02ad06156f31"
server_key_version: "encryptionkey@0"
I20260924 00:13:41.018412 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "40bdd266bc75479992e1d50ccf744bf6"
format_stamp: "Formatted at 2026-09-24 00:13:41 on dist-test-slave-3q8k"
server_key: "b6787ea39999872381a78fb3a45695c8"
server_key_iv: "8fffbd035729457ae97b02ad06156f31"
server_key_version: "encryptionkey@0"
I20260924 00:13:41.019884 21575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:41.020669 21590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:41.020844 21575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:41.020949 21575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "40bdd266bc75479992e1d50ccf744bf6"
format_stamp: "Formatted at 2026-09-24 00:13:41 on dist-test-slave-3q8k"
server_key: "b6787ea39999872381a78fb3a45695c8"
server_key_iv: "8fffbd035729457ae97b02ad06156f31"
server_key_version: "encryptionkey@0"
I20260924 00:13:41.021030 21575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:41.041427 21575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:41.043526 21575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:41.043720 21575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:41.047863 21575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37853
I20260924 00:13:41.047915 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37853 every 8 connection(s)
I20260924 00:13:41.048269 21575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:41.049168 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:41.051680 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6: Bootstrap starting.
I20260924 00:13:41.052276 21643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:41.052582 21643 log.cc:824] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:41.053269 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6: No bootstrap required, opened a new log
I20260924 00:13:41.054886 21643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 00:13:41.055030 21643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:41.055100 21643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bdd266bc75479992e1d50ccf744bf6, State: Initialized, Role: FOLLOWER
I20260924 00:13:41.055299 21643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 00:13:41.055365 21643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:41.055428 21643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:41.055480 21643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:41.056023 21643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 00:13:41.056142 21643 leader_election.cc:302] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40bdd266bc75479992e1d50ccf744bf6; no voters:
I20260924 00:13:41.056329 21643 leader_election.cc:288] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:41.056361 21646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:41.056533 21646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 40bdd266bc75479992e1d50ccf744bf6, State: Running, Role: LEADER
I20260924 00:13:41.056602 21643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:41.056666 21646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 00:13:41.057063 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40bdd266bc75479992e1d50ccf744bf6. Latest consensus state: current_term: 1 leader_uuid: "40bdd266bc75479992e1d50ccf744bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } } }
I20260924 00:13:41.057073 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40bdd266bc75479992e1d50ccf744bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40bdd266bc75479992e1d50ccf744bf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } } }
I20260924 00:13:41.057163 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:41.057341 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:41.057588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21575
I20260924 00:13:41.057706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:41.057888 392 external_mini_cluster.cc:1428] Setting key 9c525489b3b3ad09ab8da5998e7cbfe2
I20260924 00:13:41.057363 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:41.058579 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:41.060210 21650 catalog_manager.cc:1347] Generated new cluster ID: 26584c29a52745c898129115a2e6e2dc
I20260924 00:13:41.060266 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:13:41 dist-test-slave-3q8k krb5kdc[21562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:13:41.070508 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:41.058542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:36787 (local address 127.0.0.1:37853)
0924 00:13:41.059277 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:41.059285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:41.059316 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:13:41.059427 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:41.059434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:41.059502 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:41.059649 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:41.060443 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:41.061881 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:41.062928 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:41.063606 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:41.063719 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:41.066571 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:41.066576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:41.066576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:41.066589 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:41.068730 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:41.069170 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:41.069173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:41.069174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:41.069287 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:41.069520 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:41.069521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:41.069521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:41.069861 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:41.070028 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:41.070138 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:13:41.070245 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":90,"threads_started":1}
I20260924 00:13:41.071755 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:41.081269 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:41.081794 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:41.083173 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:41.073155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:53569 (local address 127.0.0.1:37853)
0924 00:13:41.073200 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:41.073202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:41.075413 (+ 2211us) server_negotiation.cc:407] Connection header received
0924 00:13:41.075484 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:41.075485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:41.075496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:41.075554 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:41.076191 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:41.076978 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:41.078029 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:41.078335 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:41.078396 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:41.079952 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:41.079953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:41.079954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:41.079961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:41.081808 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:41.082259 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:41.082260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:41.082261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:41.082316 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:41.082563 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:41.082564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:41.082564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:41.082686 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:41.082779 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:41.083014 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:13:41.083080 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:13:41.084825 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40bdd266bc75479992e1d50ccf744bf6: Generated new TSK 0
I20260924 00:13:41.085024 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:41.086773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21575
W20260924 00:13:41.091195 21560 connection.cc:535] client connection to 127.0.0.1:37853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37853): Connection reset by peer (error 104)
2026-09-24T00:13:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:41 dist-test-slave-3q8k krb5kdc[21674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:41 dist-test-slave-3q8k krb5kdc[21674](info): set up 2 sockets
Sep 24 00:13:41 dist-test-slave-3q8k krb5kdc[21674](info): commencing operation
krb5kdc: starting...
W20260924 00:13:42.155395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:42 dist-test-slave-3q8k krb5kdc[21674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:42.274667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:42.355453 21687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:42.355701 21687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:42.355742 21687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:42.356909 21687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:42.356977 21687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:42.357002 21687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:42.357023 21687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:42.357080 21687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:42.358575 21687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:42.358858 21687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:42.359107 21687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:42.364024 21692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:42.364157 21687 server_base.cc:1034] running on GCE node
W20260924 00:13:42.364038 21693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:42.364030 21695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:42.364508 21687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:42.364854 21687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:42.366035 21687 hybrid_clock.cc:648] HybridClock initialized: now 1790208822366021 us; error 29 us; skew 500 ppm
Sep 24 00:13:42 dist-test-slave-3q8k krb5kdc[21674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:42.368764 21687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:42.369307 21687 webserver.cc:460] Webserver started at http://127.0.0.1:45065/ using document root <none> and password file <none>
I20260924 00:13:42.369557 21687 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:42.369629 21687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:42.369778 21687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:42.370862 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "7e26293cc36c445fb754b8fe64442648"
format_stamp: "Formatted at 2026-09-24 00:13:42 on dist-test-slave-3q8k"
server_key: "f26a9bf66d6d4033097cdcbf8be0de25"
server_key_iv: "aeb479c1f742696bdbdad251c4335d53"
server_key_version: "encryptionkey@0"
I20260924 00:13:42.371270 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e26293cc36c445fb754b8fe64442648"
format_stamp: "Formatted at 2026-09-24 00:13:42 on dist-test-slave-3q8k"
server_key: "f26a9bf66d6d4033097cdcbf8be0de25"
server_key_iv: "aeb479c1f742696bdbdad251c4335d53"
server_key_version: "encryptionkey@0"
I20260924 00:13:42.372720 21687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:42.373508 21702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:42.373677 21687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:42.373780 21687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "7e26293cc36c445fb754b8fe64442648"
format_stamp: "Formatted at 2026-09-24 00:13:42 on dist-test-slave-3q8k"
server_key: "f26a9bf66d6d4033097cdcbf8be0de25"
server_key_iv: "aeb479c1f742696bdbdad251c4335d53"
server_key_version: "encryptionkey@0"
I20260924 00:13:42.373864 21687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:42.392939 21687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:42.394986 21687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:42.395159 21687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:42.399252 21687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43941
I20260924 00:13:42.399308 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43941 every 8 connection(s)
I20260924 00:13:42.399698 21687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:42.400698 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:42.402918 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648: Bootstrap starting.
I20260924 00:13:42.403540 21755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:42.403828 21755 log.cc:824] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:42.404518 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648: No bootstrap required, opened a new log
I20260924 00:13:42.406118 21755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260924 00:13:42.406260 21755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:42.406282 21755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e26293cc36c445fb754b8fe64442648, State: Initialized, Role: FOLLOWER
I20260924 00:13:42.406482 21755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260924 00:13:42.406567 21755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:42.406610 21755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:42.406661 21755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:42.407191 21755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260924 00:13:42.407352 21755 leader_election.cc:302] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e26293cc36c445fb754b8fe64442648; no voters:
I20260924 00:13:42.407552 21755 leader_election.cc:288] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:42.407620 21758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:42.407789 21758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [term 1 LEADER]: Becoming Leader. State: Replica: 7e26293cc36c445fb754b8fe64442648, State: Running, Role: LEADER
I20260924 00:13:42.407809 21755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:42.407882 21758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260924 00:13:42.408290 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e26293cc36c445fb754b8fe64442648. Latest consensus state: current_term: 1 leader_uuid: "7e26293cc36c445fb754b8fe64442648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } } }
I20260924 00:13:42.408428 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:42.408443 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e26293cc36c445fb754b8fe64442648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e26293cc36c445fb754b8fe64442648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } } }
I20260924 00:13:42.408618 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:42.408874 21767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:42.409420 21767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:42.409672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21687
I20260924 00:13:42.409792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:42.409981 392 external_mini_cluster.cc:1428] Setting key d840b1dc47476a192356f695a1caf40f
I20260924 00:13:42.411021 21767 catalog_manager.cc:1347] Generated new cluster ID: 21b84e1c4c0a4bf19c5df9a7ca3b6a2d
I20260924 00:13:42.411089 21767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:42.416837 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:42.410702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:34921 (local address 127.0.0.1:43941)
0924 00:13:42.411402 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:42.411411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:42.411443 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:13:42.411557 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:42.411565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:42.411624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:42.411766 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:42.411990 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:42.413404 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:42.414464 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:42.415114 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:42.415271 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:42.416018 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:42.416023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:42.416023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:42.416242 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:42.416284 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:42.416299 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:42.416391 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:42.416585 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 00:13:42.416616 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":84,"threads_started":1}
I20260924 00:13:42.417783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:42.422778 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:42.419184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:39129 (local address 127.0.0.1:43941)
0924 00:13:42.419237 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:42.419239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:42.419502 (+ 263us) server_negotiation.cc:407] Connection header received
0924 00:13:42.419532 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:42.419533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:42.419544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:42.419606 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:42.419737 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:42.420465 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:42.421413 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:42.421733 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:42.421806 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:42.422519 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:42.422521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:42.422521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:42.422526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:42.422541 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:42.422545 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:42.422604 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:42.422682 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:13:42.422695 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:13:42.437538 21767 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:42.438078 21767 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:42.441567 21767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e26293cc36c445fb754b8fe64442648: Generated new TSK 0
I20260924 00:13:42.441766 21767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:42.445655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21687
W20260924 00:13:42.449842 21672 connection.cc:535] client connection to 127.0.0.1:43941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43941): Connection reset by peer (error 104)
2026-09-24T00:13:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:42 dist-test-slave-3q8k krb5kdc[21786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:42 dist-test-slave-3q8k krb5kdc[21786](info): set up 2 sockets
Sep 24 00:13:42 dist-test-slave-3q8k krb5kdc[21786](info): commencing operation
krb5kdc: starting...
W20260924 00:13:43.479776 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:43 dist-test-slave-3q8k krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:13:43.598464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:43.680301 21799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:43.680567 21799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:43.680606 21799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:43.681824 21799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:43.681910 21799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:43.681936 21799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:43.681965 21799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:43.681994 21799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:43.683660 21799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:43.683951 21799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:43.684196 21799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:43.689182 21807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:43.689347 21799 server_base.cc:1034] running on GCE node
W20260924 00:13:43.689479 21805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:43.689184 21804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:43.689808 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:43.690119 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:43.691284 21799 hybrid_clock.cc:648] HybridClock initialized: now 1790208823691270 us; error 47 us; skew 500 ppm
Sep 24 00:13:43 dist-test-slave-3q8k krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:43.694141 21799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:13:43.694628 21799 webserver.cc:460] Webserver started at http://127.0.0.1:33887/ using document root <none> and password file <none>
I20260924 00:13:43.694868 21799 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:43.694938 21799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:43.695096 21799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:43.696272 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "c4939870045d4088bf375bbf182ba3b4"
format_stamp: "Formatted at 2026-09-24 00:13:43 on dist-test-slave-3q8k"
server_key: "fb25933eb157063bf0f215d01f239e49"
server_key_iv: "3c8808d2ad1aa5eb969dfff1673a64ab"
server_key_version: "encryptionkey@0"
I20260924 00:13:43.696672 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4939870045d4088bf375bbf182ba3b4"
format_stamp: "Formatted at 2026-09-24 00:13:43 on dist-test-slave-3q8k"
server_key: "fb25933eb157063bf0f215d01f239e49"
server_key_iv: "3c8808d2ad1aa5eb969dfff1673a64ab"
server_key_version: "encryptionkey@0"
I20260924 00:13:43.698056 21799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:43.698899 21814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:43.699113 21799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:43.699219 21799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "c4939870045d4088bf375bbf182ba3b4"
format_stamp: "Formatted at 2026-09-24 00:13:43 on dist-test-slave-3q8k"
server_key: "fb25933eb157063bf0f215d01f239e49"
server_key_iv: "3c8808d2ad1aa5eb969dfff1673a64ab"
server_key_version: "encryptionkey@0"
I20260924 00:13:43.699333 21799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:43.702953 21799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:43.705380 21799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:43.705557 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:43.709540 21799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34707
I20260924 00:13:43.709573 21866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34707 every 8 connection(s)
I20260924 00:13:43.709924 21799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:43.710871 21867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:43.713218 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: Bootstrap starting.
I20260924 00:13:43.713421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21799
I20260924 00:13:43.713585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:43.713788 392 external_mini_cluster.cc:1428] Setting key d10fb9149b7d2c11dad83ffa3509b463
I20260924 00:13:43.713904 21867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:43.714236 21867 log.cc:824] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:43.714986 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: No bootstrap required, opened a new log
I20260924 00:13:43.716322 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:43.714321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:53325 (local address 127.0.0.1:34707)
0924 00:13:43.714533 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:43.714540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:43.714565 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:13:43.714997 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:43.715003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:43.715028 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:43.715184 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:43.715857 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:43.715861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:43.715861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:43.715865 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:43.715902 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:43.715913 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:43.715989 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:43.716060 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:13:43.716121 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260924 00:13:43.717504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:43.717613 21867 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260924 00:13:43.717773 21867 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:43.717803 21867 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4939870045d4088bf375bbf182ba3b4, State: Initialized, Role: FOLLOWER
I20260924 00:13:43.717943 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260924 00:13:43.718057 21867 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:43.718091 21867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:43.718142 21867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:43.718930 21867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260924 00:13:43.719075 21867 leader_election.cc:302] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4939870045d4088bf375bbf182ba3b4; no voters:
I20260924 00:13:43.719267 21867 leader_election.cc:288] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:43.719292 21877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:43.719408 21877 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [term 1 LEADER]: Becoming Leader. State: Replica: c4939870045d4088bf375bbf182ba3b4, State: Running, Role: LEADER
I20260924 00:13:43.719540 21867 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:43.719548 21877 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260924 00:13:43.719913 21877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4939870045d4088bf375bbf182ba3b4. Latest consensus state: current_term: 1 leader_uuid: "c4939870045d4088bf375bbf182ba3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260924 00:13:43.719980 21878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4939870045d4088bf375bbf182ba3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4939870045d4088bf375bbf182ba3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260924 00:13:43.720153 21878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:43.720292 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:43.719045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:35441 (local address 127.0.0.1:34707)
0924 00:13:43.719146 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:43.719148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:43.719191 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:13:43.719313 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:43.719314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:43.719318 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:43.719365 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:43.720046 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:43.720047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:43.720048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:43.720052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:43.720066 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:43.720071 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:43.720101 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:43.720202 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:13:43.720218 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:13:43.720047 21877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4 [sys.catalog]: This master's current role is: LEADER
W20260924 00:13:43.721382 21892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:43.721454 21892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:43.721515 21886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:43.721721 21886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:43.723026 21886 catalog_manager.cc:1347] Generated new cluster ID: 186e2d36703645b2939616c38e4f1356
I20260924 00:13:43.723076 21886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:43.733067 21886 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:43.733249 21886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:43.737937 21886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4939870045d4088bf375bbf182ba3b4: Generated new TSK 0
I20260924 00:13:43.738137 21886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:43.741278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21799
2026-09-24T00:13:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:43 dist-test-slave-3q8k krb5kdc[21898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:43 dist-test-slave-3q8k krb5kdc[21898](info): set up 2 sockets
Sep 24 00:13:43 dist-test-slave-3q8k krb5kdc[21898](info): commencing operation
krb5kdc: starting...
W20260924 00:13:44.783453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:44 dist-test-slave-3q8k krb5kdc[21898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:44.906689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:44.990206 21911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:44.990414 21911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:44.990437 21911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:44.991546 21911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:44.991583 21911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:44.991597 21911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:44.991611 21911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:44.991622 21911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:44.993000 21911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:44.993180 21911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:44.993343 21911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:44.998299 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:44.998421 21919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:44.998303 21916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:44.998590 21911 server_base.cc:1034] running on GCE node
I20260924 00:13:44.998874 21911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:44.999161 21911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:45.000358 21911 hybrid_clock.cc:648] HybridClock initialized: now 1790208825000335 us; error 44 us; skew 500 ppm
Sep 24 00:13:44 dist-test-slave-3q8k krb5kdc[21898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:45.002866 21911 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:45.003372 21911 webserver.cc:460] Webserver started at http://127.0.98.62:39367/ using document root <none> and password file <none>
I20260924 00:13:45.003608 21911 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:45.003674 21911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:45.003831 21911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:45.004913 21911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "0a71e982fa52405ca39dcfd816025637"
format_stamp: "Formatted at 2026-09-24 00:13:45 on dist-test-slave-3q8k"
server_key: "d4eca0fe13cf697d2c777a0837ba755e"
server_key_iv: "3fe16953f696bc37ab8f92335b479437"
server_key_version: "encryptionkey@0"
I20260924 00:13:45.005280 21911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a71e982fa52405ca39dcfd816025637"
format_stamp: "Formatted at 2026-09-24 00:13:45 on dist-test-slave-3q8k"
server_key: "d4eca0fe13cf697d2c777a0837ba755e"
server_key_iv: "3fe16953f696bc37ab8f92335b479437"
server_key_version: "encryptionkey@0"
I20260924 00:13:45.006595 21911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:45.007432 21926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:45.007627 21911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:45.007737 21911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "0a71e982fa52405ca39dcfd816025637"
format_stamp: "Formatted at 2026-09-24 00:13:45 on dist-test-slave-3q8k"
server_key: "d4eca0fe13cf697d2c777a0837ba755e"
server_key_iv: "3fe16953f696bc37ab8f92335b479437"
server_key_version: "encryptionkey@0"
I20260924 00:13:45.007826 21911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:45.016918 21911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:45.018702 21911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:45.018872 21911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:45.022810 21911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43695
I20260924 00:13:45.022856 21978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43695 every 8 connection(s)
I20260924 00:13:45.023284 21911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:45.024118 21979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:45.026263 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637: Bootstrap starting.
I20260924 00:13:45.026777 21979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:45.027007 21979 log.cc:824] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:45.027760 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637: No bootstrap required, opened a new log
I20260924 00:13:45.029268 21979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } }
I20260924 00:13:45.029402 21979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:45.029424 21979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a71e982fa52405ca39dcfd816025637, State: Initialized, Role: FOLLOWER
I20260924 00:13:45.029558 21979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } }
I20260924 00:13:45.029633 21979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:45.029655 21979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:45.029683 21979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:45.030157 21979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } }
I20260924 00:13:45.030236 21979 leader_election.cc:302] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a71e982fa52405ca39dcfd816025637; no voters:
I20260924 00:13:45.030359 21979 leader_election.cc:288] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:45.030455 21982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:45.030609 21982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [term 1 LEADER]: Becoming Leader. State: Replica: 0a71e982fa52405ca39dcfd816025637, State: Running, Role: LEADER
I20260924 00:13:45.030634 21979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:45.030692 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } }
I20260924 00:13:45.031059 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a71e982fa52405ca39dcfd816025637. Latest consensus state: current_term: 1 leader_uuid: "0a71e982fa52405ca39dcfd816025637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } } }
I20260924 00:13:45.031147 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:45.031340 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a71e982fa52405ca39dcfd816025637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a71e982fa52405ca39dcfd816025637" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43695 } } }
I20260924 00:13:45.031431 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:45.031831 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:45.032068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 21911
I20260924 00:13:45.032217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:45.032337 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:45.032442 392 external_mini_cluster.cc:1428] Setting key fec68ad439e54357065d50221d905f74
I20260924 00:13:45.033938 21990 catalog_manager.cc:1347] Generated new cluster ID: 28f11eac1d99455791f55d2b46caaa6f
I20260924 00:13:45.033990 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:13:45 dist-test-slave-3q8k krb5kdc[21898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:45.042711 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:45.032888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38083 (local address 127.0.98.62:43695)
0924 00:13:45.033051 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:45.033061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:45.033157 (+ 96us) server_negotiation.cc:407] Connection header received
0924 00:13:45.033299 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:45.033307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:45.033361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:45.033533 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:45.034293 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:45.035183 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:45.036043 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:45.036482 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:45.036602 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:45.039492 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:45.039496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:45.039496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:45.039508 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:45.041190 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:45.041583 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:45.041586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:45.041587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:45.041682 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:45.041909 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:45.041910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:45.041910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:45.042145 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:45.042324 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:45.042417 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 00:13:45.042519 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260924 00:13:45.043637 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:45.049429 21990 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:45.050096 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:45.051146 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:45.045101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59477 (local address 127.0.98.62:43695)
0924 00:13:45.045138 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:45.045140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:45.045149 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:45.045174 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:45.045175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:45.045184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:45.045239 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:45.045733 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:45.046254 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:45.046959 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:45.047175 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:45.047267 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:45.048411 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:45.048412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:45.048412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:45.048419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:45.049909 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:45.050289 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:45.050290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:45.050291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:45.050341 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:45.050571 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:45.050572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:45.050572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:45.050681 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:45.050804 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:45.050913 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:13:45.050971 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:13:45.054555 21990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a71e982fa52405ca39dcfd816025637: Generated new TSK 0
I20260924 00:13:45.054716 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:45.057543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 21911
W20260924 00:13:45.061844 21896 connection.cc:535] client connection to 127.0.98.62:43695 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43695): Connection reset by peer (error 104)
2026-09-24T00:13:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:45 dist-test-slave-3q8k krb5kdc[22010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:45 dist-test-slave-3q8k krb5kdc[22010](info): set up 2 sockets
Sep 24 00:13:45 dist-test-slave-3q8k krb5kdc[22010](info): commencing operation
krb5kdc: starting...
W20260924 00:13:46.091455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:46 dist-test-slave-3q8k krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:46.209877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:46.290299 22023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:46.290527 22023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:46.290550 22023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:46.291814 22023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:46.291867 22023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:46.291882 22023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:46.291894 22023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:46.291937 22023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:46.293519 22023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:46.293808 22023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:46.294054 22023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:46.299136 22028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:46.299202 22031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:46.299136 22023 server_base.cc:1034] running on GCE node
W20260924 00:13:46.299173 22029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:46.299782 22023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:46.300112 22023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:46.301371 22023 hybrid_clock.cc:648] HybridClock initialized: now 1790208826301344 us; error 41 us; skew 500 ppm
Sep 24 00:13:46 dist-test-slave-3q8k krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:46.310109 22023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:46.310679 22023 webserver.cc:460] Webserver started at http://127.0.98.62:40001/ using document root <none> and password file <none>
I20260924 00:13:46.310918 22023 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:46.310992 22023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:46.311147 22023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:46.312280 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "8d48ab77964a45f2a5393c828366b2d6"
format_stamp: "Formatted at 2026-09-24 00:13:46 on dist-test-slave-3q8k"
server_key: "848303b9d4043b9966b35793bcb52755"
server_key_iv: "25fe3ac7dba99fd5c3473e2c04661831"
server_key_version: "encryptionkey@0"
I20260924 00:13:46.312708 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d48ab77964a45f2a5393c828366b2d6"
format_stamp: "Formatted at 2026-09-24 00:13:46 on dist-test-slave-3q8k"
server_key: "848303b9d4043b9966b35793bcb52755"
server_key_iv: "25fe3ac7dba99fd5c3473e2c04661831"
server_key_version: "encryptionkey@0"
I20260924 00:13:46.314109 22023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:46.314908 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:46.315121 22023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:46.315251 22023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "8d48ab77964a45f2a5393c828366b2d6"
format_stamp: "Formatted at 2026-09-24 00:13:46 on dist-test-slave-3q8k"
server_key: "848303b9d4043b9966b35793bcb52755"
server_key_iv: "25fe3ac7dba99fd5c3473e2c04661831"
server_key_version: "encryptionkey@0"
I20260924 00:13:46.315343 22023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:46.328008 22023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:46.330370 22023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:46.330582 22023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:46.334810 22023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42941
I20260924 00:13:46.334837 22090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42941 every 8 connection(s)
I20260924 00:13:46.335192 22023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:46.336131 22091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:46.338446 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6: Bootstrap starting.
I20260924 00:13:46.339051 22091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:46.339385 22091 log.cc:824] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:46.340112 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6: No bootstrap required, opened a new log
I20260924 00:13:46.341760 22091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } }
I20260924 00:13:46.341912 22091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:46.341938 22091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d48ab77964a45f2a5393c828366b2d6, State: Initialized, Role: FOLLOWER
I20260924 00:13:46.342129 22091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } }
I20260924 00:13:46.342216 22091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:46.342239 22091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:46.342288 22091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:46.342830 22091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } }
I20260924 00:13:46.342944 22091 leader_election.cc:302] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d48ab77964a45f2a5393c828366b2d6; no voters:
I20260924 00:13:46.343135 22091 leader_election.cc:288] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:46.343180 22094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:46.343436 22094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d48ab77964a45f2a5393c828366b2d6, State: Running, Role: LEADER
I20260924 00:13:46.343436 22091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:46.343570 22094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } }
I20260924 00:13:46.343991 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d48ab77964a45f2a5393c828366b2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } } }
I20260924 00:13:46.344072 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:46.344120 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d48ab77964a45f2a5393c828366b2d6. Latest consensus state: current_term: 1 leader_uuid: "8d48ab77964a45f2a5393c828366b2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48ab77964a45f2a5393c828366b2d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42941 } } }
I20260924 00:13:46.344182 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:46.344336 22102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:46.345034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22023
I20260924 00:13:46.345093 22102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:46.345165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:46.345356 392 external_mini_cluster.cc:1428] Setting key aea92993fe2e11b34c997db9969f0d7f
I20260924 00:13:46.346828 22102 catalog_manager.cc:1347] Generated new cluster ID: 910c03dcf9b7452d9932bddade784c74
I20260924 00:13:46.346889 22102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:46.352249 22102 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:46.352636 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:46.346000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50453 (local address 127.0.98.62:42941)
0924 00:13:46.347617 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:46.347626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:46.347653 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:13:46.347767 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:46.347783 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:46.347831 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:46.347988 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:46.348277 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:46.349342 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:46.350288 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:46.350766 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:46.350922 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:46.351788 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:46.351792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:46.351793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:46.352023 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:46.352066 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:46.352082 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:46.352215 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:46.352393 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 00:13:46.352428 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":70,"threads_started":1}
I20260924 00:13:46.353147 22102 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:46.353691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:46.358235 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:46.355094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.0.98.62:42941)
0924 00:13:46.355131 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:46.355134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:46.355249 (+ 115us) server_negotiation.cc:407] Connection header received
0924 00:13:46.355325 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:46.355326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:46.355336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:46.355388 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:46.355546 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:46.356074 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:46.356818 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:46.357071 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:46.357146 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:46.357942 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:46.357943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:46.357943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:46.357948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:46.357975 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:46.357979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:46.358055 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:46.358120 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 00:13:46.358133 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:46.362993 22102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d48ab77964a45f2a5393c828366b2d6: Generated new TSK 0
I20260924 00:13:46.363209 22102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:46.368813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22023
W20260924 00:13:46.373070 22008 connection.cc:535] client connection to 127.0.98.62:42941 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42941): Connection reset by peer (error 104)
2026-09-24T00:13:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:46 dist-test-slave-3q8k krb5kdc[22122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:46 dist-test-slave-3q8k krb5kdc[22122](info): set up 2 sockets
Sep 24 00:13:46 dist-test-slave-3q8k krb5kdc[22122](info): commencing operation
krb5kdc: starting...
W20260924 00:13:47.399381 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:47 dist-test-slave-3q8k krb5kdc[22122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:47.523198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:47.604097 22135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:47.604310 22135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:47.604333 22135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:47.605512 22135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:47.605561 22135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:47.605575 22135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:47.605587 22135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:47.605599 22135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:47.607102 22135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:47.607394 22135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:47.607643 22135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:47.612550 22143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:47.612571 22141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:47.612792 22140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:47.612673 22135 server_base.cc:1034] running on GCE node
I20260924 00:13:47.613059 22135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:47.613379 22135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:47.614550 22135 hybrid_clock.cc:648] HybridClock initialized: now 1790208827614529 us; error 40 us; skew 500 ppm
Sep 24 00:13:47 dist-test-slave-3q8k krb5kdc[22122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:47.617062 22135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:47.617502 22135 webserver.cc:460] Webserver started at http://127.0.98.62:38523/ using document root <none> and password file <none>
I20260924 00:13:47.617743 22135 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:47.617810 22135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:47.617969 22135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:47.619030 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "e557fb0081cb483e851c2de1ab16ed59"
format_stamp: "Formatted at 2026-09-24 00:13:47 on dist-test-slave-3q8k"
server_key: "789c2c719c4e921a0c16a899be673781"
server_key_iv: "897d8d3684f26c94cec97126d4bd0713"
server_key_version: "encryptionkey@0"
I20260924 00:13:47.619410 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "e557fb0081cb483e851c2de1ab16ed59"
format_stamp: "Formatted at 2026-09-24 00:13:47 on dist-test-slave-3q8k"
server_key: "789c2c719c4e921a0c16a899be673781"
server_key_iv: "897d8d3684f26c94cec97126d4bd0713"
server_key_version: "encryptionkey@0"
I20260924 00:13:47.620711 22135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:47.621470 22150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:47.621630 22135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:47.621726 22135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "e557fb0081cb483e851c2de1ab16ed59"
format_stamp: "Formatted at 2026-09-24 00:13:47 on dist-test-slave-3q8k"
server_key: "789c2c719c4e921a0c16a899be673781"
server_key_iv: "897d8d3684f26c94cec97126d4bd0713"
server_key_version: "encryptionkey@0"
I20260924 00:13:47.621806 22135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:47.625113 22135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:47.626916 22135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:47.627076 22135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:47.630863 22135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36381
I20260924 00:13:47.630908 22202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36381 every 8 connection(s)
I20260924 00:13:47.631310 22135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:47.632321 22203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:47.634603 22203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59: Bootstrap starting.
I20260924 00:13:47.635190 22203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:47.635540 22203 log.cc:824] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:47.636255 22203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59: No bootstrap required, opened a new log
I20260924 00:13:47.637836 22203 raft_consensus.cc:348] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } }
I20260924 00:13:47.638065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22135
I20260924 00:13:47.638093 22203 raft_consensus.cc:374] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:47.638135 22203 raft_consensus.cc:729] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e557fb0081cb483e851c2de1ab16ed59, State: Initialized, Role: FOLLOWER
I20260924 00:13:47.638172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:47.638288 22203 consensus_queue.cc:260] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } }
I20260924 00:13:47.638352 392 external_mini_cluster.cc:1428] Setting key 52b6065bb664b830263c82b3944d1dab
I20260924 00:13:47.638371 22203 raft_consensus.cc:388] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:47.638405 22203 raft_consensus.cc:482] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:47.638458 22203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:47.639614 22203 raft_consensus.cc:504] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } }
I20260924 00:13:47.639739 22203 leader_election.cc:302] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e557fb0081cb483e851c2de1ab16ed59; no voters:
I20260924 00:13:47.639915 22203 leader_election.cc:288] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:47.639952 22208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:47.640164 22208 raft_consensus.cc:686] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [term 1 LEADER]: Becoming Leader. State: Replica: e557fb0081cb483e851c2de1ab16ed59, State: Running, Role: LEADER
I20260924 00:13:47.640204 22203 sys_catalog.cc:564] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:47.640322 22208 consensus_queue.cc:237] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } }
I20260924 00:13:47.640753 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e557fb0081cb483e851c2de1ab16ed59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } } }
I20260924 00:13:47.640813 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:47.638881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47745 (local address 127.0.98.62:36381)
0924 00:13:47.639088 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:47.639100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:47.639126 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:13:47.639312 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:47.639321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:47.639355 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:47.639481 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:47.640264 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:47.640268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:47.640269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:47.640276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:47.640328 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:47.640344 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:47.640398 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:47.640559 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 00:13:47.640605 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":120,"threads_started":1}
I20260924 00:13:47.640830 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:47.641068 22215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:47.641671 22215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:47.641693 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e557fb0081cb483e851c2de1ab16ed59. Latest consensus state: current_term: 1 leader_uuid: "e557fb0081cb483e851c2de1ab16ed59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e557fb0081cb483e851c2de1ab16ed59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36381 } } }
I20260924 00:13:47.641992 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:47.641840 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:47.643627 22215 catalog_manager.cc:1347] Generated new cluster ID: 72af7f3efd0f409babc2d8a4eda7bcab
I20260924 00:13:47.643680 22215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:47.644706 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:47.643457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41087 (local address 127.0.98.62:36381)
0924 00:13:47.643564 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:47.643566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:47.643644 (+ 78us) server_negotiation.cc:407] Connection header received
0924 00:13:47.643758 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:47.643758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:47.643764 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:47.643837 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:47.644454 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:47.644455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:47.644455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:47.644460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:47.644474 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:47.644478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:47.644536 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:47.644595 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 00:13:47.644610 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 00:13:47.665417 22215 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:47.665630 22215 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:47.671449 22215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e557fb0081cb483e851c2de1ab16ed59: Generated new TSK 0
I20260924 00:13:47.671679 22215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:47.678611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22135
2026-09-24T00:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:47 dist-test-slave-3q8k krb5kdc[22234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:47 dist-test-slave-3q8k krb5kdc[22234](info): set up 2 sockets
Sep 24 00:13:47 dist-test-slave-3q8k krb5kdc[22234](info): commencing operation
krb5kdc: starting...
W20260924 00:13:48.703399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:48 dist-test-slave-3q8k krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:48.824155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:48.909037 22247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:48.909271 22247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:48.909318 22247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:48.910467 22247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:48.910534 22247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:48.910561 22247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:48.910589 22247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:48.910619 22247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:48.912150 22247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:48.912421 22247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:48.912647 22247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:48.917593 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:48.917650 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:48.917608 22255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:48.917789 22247 server_base.cc:1034] running on GCE node
I20260924 00:13:48.918098 22247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:48.918438 22247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:48.919591 22247 hybrid_clock.cc:648] HybridClock initialized: now 1790208828919581 us; error 51 us; skew 500 ppm
Sep 24 00:13:48 dist-test-slave-3q8k krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:48.921959 22247 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:48.922372 22247 webserver.cc:460] Webserver started at http://127.0.98.62:35687/ using document root <none> and password file <none>
I20260924 00:13:48.922610 22247 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:48.922680 22247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:48.922835 22247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:48.923976 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "0aa02fff8f8d4eae97a1f45541fe6363"
format_stamp: "Formatted at 2026-09-24 00:13:48 on dist-test-slave-3q8k"
server_key: "ab89fb6897e037910475a90a9e14ec27"
server_key_iv: "dc369b8a3edf46fc39f0068a9468a23e"
server_key_version: "encryptionkey@0"
I20260924 00:13:48.924336 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "0aa02fff8f8d4eae97a1f45541fe6363"
format_stamp: "Formatted at 2026-09-24 00:13:48 on dist-test-slave-3q8k"
server_key: "ab89fb6897e037910475a90a9e14ec27"
server_key_iv: "dc369b8a3edf46fc39f0068a9468a23e"
server_key_version: "encryptionkey@0"
I20260924 00:13:48.925657 22247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:13:48.926466 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:48.926647 22247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:48.926756 22247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "0aa02fff8f8d4eae97a1f45541fe6363"
format_stamp: "Formatted at 2026-09-24 00:13:48 on dist-test-slave-3q8k"
server_key: "ab89fb6897e037910475a90a9e14ec27"
server_key_iv: "dc369b8a3edf46fc39f0068a9468a23e"
server_key_version: "encryptionkey@0"
I20260924 00:13:48.926846 22247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:48.939913 22247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:48.941718 22247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:48.941883 22247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:48.945837 22247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43003
I20260924 00:13:48.945868 22314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43003 every 8 connection(s)
I20260924 00:13:48.946208 22247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:48.947058 22315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:48.948912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22247
I20260924 00:13:48.949036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:48.949220 392 external_mini_cluster.cc:1428] Setting key 81a3d142bdca1dbb2e5f8320b43ec60d
I20260924 00:13:48.949306 22315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: Bootstrap starting.
I20260924 00:13:48.949981 22315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:48.950273 22315 log.cc:824] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:48.950929 22315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: No bootstrap required, opened a new log
I20260924 00:13:48.952673 22315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } }
I20260924 00:13:48.952821 22315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:48.952890 22315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02fff8f8d4eae97a1f45541fe6363, State: Initialized, Role: FOLLOWER
I20260924 00:13:48.953042 22315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } }
I20260924 00:13:48.953119 22315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:48.953162 22315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:48.953209 22315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:48.953707 22315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } }
I20260924 00:13:48.953819 22315 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa02fff8f8d4eae97a1f45541fe6363; no voters:
I20260924 00:13:48.953999 22315 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:48.954072 22320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:48.954241 22315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:48.955197 22320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa02fff8f8d4eae97a1f45541fe6363, State: Running, Role: LEADER
I20260924 00:13:48.955345 22320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } }
Sep 24 00:13:48 dist-test-slave-3q8k krb5kdc[22234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:48.955866 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } } }
I20260924 00:13:48.955885 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa02fff8f8d4eae97a1f45541fe6363. Latest consensus state: current_term: 1 leader_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa02fff8f8d4eae97a1f45541fe6363" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43003 } } }
I20260924 00:13:48.955948 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:48.955978 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363 [sys.catalog]: This master's current role is: LEADER
W20260924 00:13:48.956496 22334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:48.956547 22334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:48.956686 22335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:48.956902 22335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:48.958678 22335 catalog_manager.cc:1347] Generated new cluster ID: 6e877a6c00294e3eabe04fcebabb41b4
I20260924 00:13:48.958734 22335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:48.960453 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:48.949736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48463 (local address 127.0.98.62:43003)
0924 00:13:48.949927 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:48.949934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:48.949957 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:13:48.950515 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:48.950522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:48.950567 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:48.950709 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:48.951465 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:48.952330 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:48.953234 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:48.953657 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:48.953776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:48.956683 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:48.956687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:48.956687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:48.956698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:48.958575 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:48.958963 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:48.958966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:48.958966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:48.959080 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:48.959285 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:48.959286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:48.959286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:48.959562 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:48.959692 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:48.960124 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 00:13:48.960243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 00:13:48.968325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:48.970378 22335 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:48.970873 22335 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:48.974471 22335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa02fff8f8d4eae97a1f45541fe6363: Generated new TSK 0
I20260924 00:13:48.974696 22335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:48.977140 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:48.969813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44203 (local address 127.0.98.62:43003)
0924 00:13:48.969853 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:48.969855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:48.969997 (+ 142us) server_negotiation.cc:407] Connection header received
0924 00:13:48.970204 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:48.970205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:48.970213 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:48.970286 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:48.970948 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:48.971678 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:48.972701 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:48.972998 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:48.973055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:48.974595 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:48.974596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:48.974596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:48.974603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:48.976061 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:48.976475 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:48.976476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:48.976476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:48.976528 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:48.976721 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:48.976722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:48.976722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:48.976812 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:48.976880 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:48.977020 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 00:13:48.977085 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:48.979352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22247
2026-09-24T00:13:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:49 dist-test-slave-3q8k krb5kdc[22346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:49 dist-test-slave-3q8k krb5kdc[22346](info): set up 2 sockets
Sep 24 00:13:49 dist-test-slave-3q8k krb5kdc[22346](info): commencing operation
krb5kdc: starting...
W20260924 00:13:50.043529 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:50 dist-test-slave-3q8k krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:50.166637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:50.246484 22359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:50.246687 22359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:50.246709 22359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:50.247780 22359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:50.247817 22359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:50.247831 22359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:50.247843 22359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:50.247855 22359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:50.249223 22359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:50.249415 22359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:50.249576 22359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:50.254534 22365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:50.254558 22367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:50.254635 22359 server_base.cc:1034] running on GCE node
W20260924 00:13:50.254558 22364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:50.255031 22359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:50.255383 22359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:50.256556 22359 hybrid_clock.cc:648] HybridClock initialized: now 1790208830256537 us; error 31 us; skew 500 ppm
Sep 24 00:13:50 dist-test-slave-3q8k krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:50.259042 22359 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:50.259567 22359 webserver.cc:460] Webserver started at http://127.0.98.62:36159/ using document root <none> and password file <none>
I20260924 00:13:50.259799 22359 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:50.259866 22359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:50.260025 22359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:50.261179 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "a349e86400364569bb283dd20b8374de"
format_stamp: "Formatted at 2026-09-24 00:13:50 on dist-test-slave-3q8k"
server_key: "d186fe66eae253314ecf46cc0a6ecf88"
server_key_iv: "29857aecb05c34fdc3f60c323f2144f9"
server_key_version: "encryptionkey@0"
I20260924 00:13:50.261538 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "a349e86400364569bb283dd20b8374de"
format_stamp: "Formatted at 2026-09-24 00:13:50 on dist-test-slave-3q8k"
server_key: "d186fe66eae253314ecf46cc0a6ecf88"
server_key_iv: "29857aecb05c34fdc3f60c323f2144f9"
server_key_version: "encryptionkey@0"
I20260924 00:13:50.262861 22359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:50.263655 22374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:50.263862 22359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:13:50.263955 22359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "a349e86400364569bb283dd20b8374de"
format_stamp: "Formatted at 2026-09-24 00:13:50 on dist-test-slave-3q8k"
server_key: "d186fe66eae253314ecf46cc0a6ecf88"
server_key_iv: "29857aecb05c34fdc3f60c323f2144f9"
server_key_version: "encryptionkey@0"
I20260924 00:13:50.264012 22359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:50.277138 22359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:50.279536 22359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:50.279654 22359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:50.283483 22359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35883
I20260924 00:13:50.283561 22426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35883 every 8 connection(s)
I20260924 00:13:50.283877 22359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:50.284775 22427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:50.287026 22427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de: Bootstrap starting.
I20260924 00:13:50.287623 22427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:50.287915 22427 log.cc:824] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:50.288602 22427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de: No bootstrap required, opened a new log
I20260924 00:13:50.290170 22427 raft_consensus.cc:348] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } }
I20260924 00:13:50.290341 22427 raft_consensus.cc:374] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:50.290391 22427 raft_consensus.cc:729] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a349e86400364569bb283dd20b8374de, State: Initialized, Role: FOLLOWER
I20260924 00:13:50.290557 22427 consensus_queue.cc:260] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } }
I20260924 00:13:50.290655 22427 raft_consensus.cc:388] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:50.290699 22427 raft_consensus.cc:482] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:50.290747 22427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:50.291333 22427 raft_consensus.cc:504] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } }
I20260924 00:13:50.291424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22359
I20260924 00:13:50.291452 22427 leader_election.cc:302] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a349e86400364569bb283dd20b8374de; no voters:
I20260924 00:13:50.291553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:50.291637 22427 leader_election.cc:288] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:50.291680 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:50.291755 392 external_mini_cluster.cc:1428] Setting key fbacd44cc0c8791b64e56ce62044e5a2
I20260924 00:13:50.291849 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [term 1 LEADER]: Becoming Leader. State: Replica: a349e86400364569bb283dd20b8374de, State: Running, Role: LEADER
I20260924 00:13:50.291893 22427 sys_catalog.cc:564] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:50.291966 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } }
I20260924 00:13:50.292343 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a349e86400364569bb283dd20b8374de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } } }
I20260924 00:13:50.292379 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [sys.catalog]: SysCatalogTable state changed. Reason: New leader a349e86400364569bb283dd20b8374de. Latest consensus state: current_term: 1 leader_uuid: "a349e86400364569bb283dd20b8374de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a349e86400364569bb283dd20b8374de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35883 } } }
I20260924 00:13:50.292544 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:50.292752 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:50.293219 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:50.293732 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:50.295397 22436 catalog_manager.cc:1347] Generated new cluster ID: d420895c9bd84060b8f36f89cab3d72f
I20260924 00:13:50.295449 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:50.298121 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:50.292246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48287 (local address 127.0.98.62:35883)
0924 00:13:50.292531 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:50.292541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:50.292576 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:13:50.292693 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:50.292700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:50.292789 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:50.292951 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:50.293358 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:50.294704 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:50.295947 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:50.296577 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:50.296675 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:50.297430 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:50.297435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:50.297435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:50.297652 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:50.297688 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:50.297702 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:50.297762 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:50.297857 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:13:50.297924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260924 00:13:50.298990 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:50.303954 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:50.300673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41139 (local address 127.0.98.62:35883)
0924 00:13:50.300716 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:50.300718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:50.300724 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:13:50.300795 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:50.300795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:50.300804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:50.300830 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:50.300997 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:50.301720 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:50.302688 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:50.302971 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:50.303006 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:50.303722 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:50.303723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:50.303723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:50.303727 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:50.303740 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:50.303743 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:50.303768 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:50.303858 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 00:13:50.303882 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:13:50.323041 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:50.323807 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:50.329073 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a349e86400364569bb283dd20b8374de: Generated new TSK 0
I20260924 00:13:50.329268 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:50.339167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22359
2026-09-24T00:13:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:50 dist-test-slave-3q8k krb5kdc[22458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:50 dist-test-slave-3q8k krb5kdc[22458](info): set up 2 sockets
Sep 24 00:13:50 dist-test-slave-3q8k krb5kdc[22458](info): commencing operation
krb5kdc: starting...
W20260924 00:13:51.387554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:51 dist-test-slave-3q8k krb5kdc[22458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:51.531024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38587
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:51.614148 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:51.614383 22471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:51.614408 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:51.615803 22471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:51.615863 22471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:51.615880 22471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:51.615892 22471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:51.615967 22471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:51.617616 22471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:51.617898 22471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:51.618152 22471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:51.623164 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:51.623262 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:51.623167 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:51.623541 22471 server_base.cc:1034] running on GCE node
I20260924 00:13:51.623926 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:51.624246 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:51.625442 22471 hybrid_clock.cc:648] HybridClock initialized: now 1790208831625416 us; error 38 us; skew 500 ppm
Sep 24 00:13:51 dist-test-slave-3q8k krb5kdc[22458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:51.628050 22471 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:51.628602 22471 webserver.cc:460] Webserver started at http://127.0.98.62:45421/ using document root <none> and password file <none>
I20260924 00:13:51.628849 22471 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:51.628919 22471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:51.629072 22471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:51.630294 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "deb0e24805d34fc3a17c28fd06eda782"
format_stamp: "Formatted at 2026-09-24 00:13:51 on dist-test-slave-3q8k"
server_key: "0f34f48fbce25f0d54f05f491b0439aa"
server_key_iv: "3b35801c55a329556656f32c787d922a"
server_key_version: "encryptionkey@0"
I20260924 00:13:51.630666 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "deb0e24805d34fc3a17c28fd06eda782"
format_stamp: "Formatted at 2026-09-24 00:13:51 on dist-test-slave-3q8k"
server_key: "0f34f48fbce25f0d54f05f491b0439aa"
server_key_iv: "3b35801c55a329556656f32c787d922a"
server_key_version: "encryptionkey@0"
I20260924 00:13:51.632143 22471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:13:51.632921 22486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:51.633087 22471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:13:51.633181 22471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "deb0e24805d34fc3a17c28fd06eda782"
format_stamp: "Formatted at 2026-09-24 00:13:51 on dist-test-slave-3q8k"
server_key: "0f34f48fbce25f0d54f05f491b0439aa"
server_key_iv: "3b35801c55a329556656f32c787d922a"
server_key_version: "encryptionkey@0"
I20260924 00:13:51.633273 22471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:51.636744 22471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:51.639093 22471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:51.639320 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:51.643697 22471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38587
I20260924 00:13:51.643725 22538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38587 every 8 connection(s)
I20260924 00:13:51.644114 22471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:51.645097 22539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:51.646190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22471
I20260924 00:13:51.646312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:51.646492 392 external_mini_cluster.cc:1428] Setting key 251edea596c875277eda7563312e1380
I20260924 00:13:51.648693 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782: Bootstrap starting.
I20260924 00:13:51.649013 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:51.647066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36645 (local address 127.0.98.62:38587)
0924 00:13:51.647268 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:51.647278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:51.647315 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:13:51.647729 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:51.647735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:51.647759 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:51.647874 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:51.648561 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:51.648565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:51.648565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:51.648569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:51.648607 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:51.648618 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:51.648680 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:51.648812 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:13:51.648858 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 00:13:51.649433 22539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:51.649919 22539 log.cc:824] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:51.650383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:51.650710 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782: No bootstrap required, opened a new log
I20260924 00:13:51.652983 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:51.651869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60497 (local address 127.0.98.62:38587)
0924 00:13:51.651904 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:51.651906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:51.652027 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:13:51.652102 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:51.652102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:51.652107 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:51.652153 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:51.652779 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:51.652780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:51.652780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:13:51.652785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:51.652799 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:13:51.652804 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:51.652850 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:51.652904 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:13:51.652918 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 00:13:51.652890 22539 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } }
I20260924 00:13:51.653087 22539 raft_consensus.cc:374] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:13:51.653126 22539 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb0e24805d34fc3a17c28fd06eda782, State: Initialized, Role: FOLLOWER
I20260924 00:13:51.653266 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } }
I20260924 00:13:51.653342 22539 raft_consensus.cc:388] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:13:51.653388 22539 raft_consensus.cc:482] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:13:51.653448 22539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:51.654023 22539 raft_consensus.cc:504] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } }
I20260924 00:13:51.654110 22539 leader_election.cc:302] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deb0e24805d34fc3a17c28fd06eda782; no voters:
I20260924 00:13:51.654336 22539 leader_election.cc:288] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:13:51.654424 22549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:51.654587 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:51.654592 22549 raft_consensus.cc:686] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [term 1 LEADER]: Becoming Leader. State: Replica: deb0e24805d34fc3a17c28fd06eda782, State: Running, Role: LEADER
I20260924 00:13:51.654698 22549 consensus_queue.cc:237] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } }
I20260924 00:13:51.655093 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deb0e24805d34fc3a17c28fd06eda782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } } }
I20260924 00:13:51.655205 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:51.655114 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deb0e24805d34fc3a17c28fd06eda782. Latest consensus state: current_term: 1 leader_uuid: "deb0e24805d34fc3a17c28fd06eda782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deb0e24805d34fc3a17c28fd06eda782" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38587 } } }
I20260924 00:13:51.655644 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:51.655575 22558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:51.656136 22558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:51.657795 22558 catalog_manager.cc:1347] Generated new cluster ID: 8a415cbae36447cbad33ce53bf062e58
I20260924 00:13:51.657846 22558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:51.666850 22558 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:51.667073 22558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:51.673112 22558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deb0e24805d34fc3a17c28fd06eda782: Generated new TSK 0
I20260924 00:13:51.673341 22558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:13:51.684334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22471
2026-09-24T00:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1344 ms)
[----------] 18 tests from AuthTokenIssuingTest (23899 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:51 dist-test-slave-3q8k krb5kdc[22570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:51 dist-test-slave-3q8k krb5kdc[22570](info): set up 2 sockets
Sep 24 00:13:51 dist-test-slave-3q8k krb5kdc[22570](info): commencing operation
krb5kdc: starting...
W20260924 00:13:52.719525 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:52 dist-test-slave-3q8k krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:52.846946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:52.932124 22583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:52.932348 22583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:52.932366 22583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:52.932387 22583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:52.933463 22583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:52.933499 22583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:52.933514 22583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:52.933526 22583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:52.933538 22583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:52.935012 22583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:52.935313 22583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:52.935599 22583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:52.940891 22591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:52.940896 22589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:52.940908 22588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:52.941262 22583 server_base.cc:1034] running on GCE node
I20260924 00:13:52.941545 22583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:52.941798 22583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:52.942945 22583 hybrid_clock.cc:648] HybridClock initialized: now 1790208832942929 us; error 31 us; skew 500 ppm
Sep 24 00:13:52 dist-test-slave-3q8k krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:52.945704 22583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:52.946256 22583 webserver.cc:460] Webserver started at http://127.0.98.62:39423/ using document root <none> and password file <none>
I20260924 00:13:52.946532 22583 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:52.946625 22583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:52.946797 22583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:52.948004 22583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "1882f3b7f0894fe3909bd51acb63ce0f"
format_stamp: "Formatted at 2026-09-24 00:13:52 on dist-test-slave-3q8k"
server_key: "440a593a5657ef658d3de62d192bfc1c"
server_key_iv: "e2ca27f031354cea0b4624b519b969c0"
server_key_version: "encryptionkey@0"
I20260924 00:13:52.948393 22583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "1882f3b7f0894fe3909bd51acb63ce0f"
format_stamp: "Formatted at 2026-09-24 00:13:52 on dist-test-slave-3q8k"
server_key: "440a593a5657ef658d3de62d192bfc1c"
server_key_iv: "e2ca27f031354cea0b4624b519b969c0"
server_key_version: "encryptionkey@0"
I20260924 00:13:52.949816 22583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:13:52.950645 22598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:52.950841 22583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:52.950958 22583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "1882f3b7f0894fe3909bd51acb63ce0f"
format_stamp: "Formatted at 2026-09-24 00:13:52 on dist-test-slave-3q8k"
server_key: "440a593a5657ef658d3de62d192bfc1c"
server_key_iv: "e2ca27f031354cea0b4624b519b969c0"
server_key_version: "encryptionkey@0"
I20260924 00:13:52.951038 22583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:52.968495 22583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:52.970659 22583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:52.970861 22583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:52.975363 22583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34037
I20260924 00:13:52.975404 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34037 every 8 connection(s)
I20260924 00:13:52.975785 22583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:52.976861 22651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:52.978639 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:52.982095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22583
I20260924 00:13:52.982293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:52.982520 392 external_mini_cluster.cc:1428] Setting key 6e2073107c7dc54fa717cc073301d636
Sep 24 00:13:52 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:52.989964 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:52.979367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.0.1:40290)
0924 00:13:52.979528 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:52.979590 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:52.980069 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:52.980428 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:52.980430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:52.980747 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:52.980900 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:52.980904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:52.981974 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:52.981975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:52.982787 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:52.982790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:52.983330 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:52.983331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:52.983395 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:52.984060 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:52.984070 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:52.985883 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:52.988369 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:52.988376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:52.988378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:52.988663 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:52.988966 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:52.988967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:52.988967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:52.989056 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:52.989531 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:52.989536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:52.989595 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:13:52.989667 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:13:52.989800 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260924 00:13:52.990049 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:52.979523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.0.98.62:34037)
0924 00:13:52.979642 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:52.979646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:52.979682 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:13:52.980302 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:52.980304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:52.980321 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:52.980424 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:52.981011 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:52.981852 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:52.982870 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:52.983172 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:52.983329 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:52.985981 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:52.985990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:52.985990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:52.985999 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:52.988231 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:52.988749 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:52.988754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:52.988754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:52.988867 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:52.989131 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:52.989132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:52.989132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:52.989450 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:52.989545 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:52.989728 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:13:52.989887 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":92,"thread_start_us":47,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 00:13:52.990823 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:52.991185 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:52.991065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34228)
0924 00:13:52.991113 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:52.991167 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:13:52.991400 22599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
I20260924 00:13:52.991876 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:52.991782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34240)
0924 00:13:52.991831 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:52.991859 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:13:52.992100 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:13:53.011569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34745
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:53.041314 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } attempt: 1
I20260924 00:13:53.041875 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.041720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34256)
0924 00:13:53.041806 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.041842 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 00:13:53.042249 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.042145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34264)
0924 00:13:53.042199 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.042231 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:13:53.042337 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
W20260924 00:13:53.093160 22658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:53.093442 22658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:53.093467 22658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:53.093498 22658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:53.094622 22658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:53.094667 22658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:53.094682 22658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:53.094695 22658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:53.094707 22658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:53.096220 22658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:53.096475 22658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:53.096724 22658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:53.101526 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:53.101495 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:53.101495 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:53.101818 22658 server_base.cc:1034] running on GCE node
I20260924 00:13:53.102097 22658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:53.102406 22658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:53.103578 22658 hybrid_clock.cc:648] HybridClock initialized: now 1790208833103547 us; error 65 us; skew 500 ppm
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:53.106098 22658 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:13:53.106602 22658 webserver.cc:460] Webserver started at http://127.0.98.61:34399/ using document root <none> and password file <none>
I20260924 00:13:53.106856 22658 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:53.106935 22658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:53.107093 22658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:53.108280 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data/instance:
uuid: "dcc9c533349445c1b638cc7ac49d3a55"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "c11e93ecc83e886bd1200f14bb8bae4f"
server_key_iv: "959ddb2802ae10b11e476631ba2f1710"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.108656 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal/instance:
uuid: "dcc9c533349445c1b638cc7ac49d3a55"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "c11e93ecc83e886bd1200f14bb8bae4f"
server_key_iv: "959ddb2802ae10b11e476631ba2f1710"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.109967 22658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:13:53.110471 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } attempt: 2
I20260924 00:13:53.110857 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:53.110948 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.110802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34266)
0924 00:13:53.110872 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.110911 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 00:13:53.111096 22658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:13:53.111352 22658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "dcc9c533349445c1b638cc7ac49d3a55"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "c11e93ecc83e886bd1200f14bb8bae4f"
server_key_iv: "959ddb2802ae10b11e476631ba2f1710"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.111423 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.111277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34282)
0924 00:13:53.111365 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.111398 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 00:13:53.111444 22658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260924 00:13:53.111514 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34745: connect: Connection refused (error 111)
I20260924 00:13:53.119704 22658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:53.121537 22658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:53.121647 22658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:53.125828 22658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34745
I20260924 00:13:53.125910 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34745 every 8 connection(s)
I20260924 00:13:53.126236 22658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:13:53.126389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22658
I20260924 00:13:53.126498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-1/wal/instance
I20260924 00:13:53.126727 392 external_mini_cluster.cc:1428] Setting key eb34b9c6e214a241fb0a253e91a18465
I20260924 00:13:53.127303 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:53.129305 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:53.142740 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.130245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.98.62:34037)
0924 00:13:53.130297 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.130300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.130466 (+ 166us) server_negotiation.cc:407] Connection header received
0924 00:13:53.131269 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.131270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.131283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.131350 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.132298 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.132829 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.134375 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.134590 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.134652 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.139434 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.139436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.139436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.139443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.141133 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.141685 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.141686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.141687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.141756 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.142050 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.142051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.142051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.142177 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.142257 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.142545 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.142641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:13:53.142951 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.130100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.0.1:40300)
0924 00:13:53.130312 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.130361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.131061 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.131450 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.131454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.131873 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.132177 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.132184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.132931 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.132932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.134275 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.134278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.134721 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.134723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.134870 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.135499 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.135510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.139323 (+ 3813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.141285 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.141293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.141295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.141556 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.141847 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.141848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.141849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.141970 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.142249 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.142256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.142409 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.142524 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.142694 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260924 00:13:53.143613 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:53.152433 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.144007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.0.98.61:34745)
0924 00:13:53.144172 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.144176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.144192 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:13:53.144209 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.144210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.144222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.144346 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.144627 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.145327 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.146012 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.146333 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.146424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.149392 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.149402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.149403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.149410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.151045 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.151505 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.151508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.151509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.151573 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.151786 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.151787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.151787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.151943 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.152035 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.152190 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.152301 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260924 00:13:53.152439 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.143929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34288)
0924 00:13:53.143993 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.144003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.144048 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.144349 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.144350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.144501 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.144550 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.144551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.145411 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.145412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.145927 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.145929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.146424 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.146425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.146489 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.146938 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.146943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.149280 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.151186 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.151187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.151188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.151408 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.151645 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.151646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.151647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.151701 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.152022 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.152023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.152062 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.152160 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.152353 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":97}
I20260924 00:13:53.153170 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:53.153615 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.153494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58876)
0924 00:13:53.153545 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.153592 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 00:13:53.153774 22677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:13:53.154296 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.154200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58892)
0924 00:13:53.154244 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.154268 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:13:53.154414 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
I20260924 00:13:53.154606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41471
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:53.202648 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } attempt: 3
I20260924 00:13:53.203673 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } attempt: 1
I20260924 00:13:53.204099 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.203955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58902)
0924 00:13:53.204015 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.204059 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:13:53.204514 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.204317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58912)
0924 00:13:53.204465 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.204487 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260924 00:13:53.204596 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:53.210461 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.203060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34300)
0924 00:13:53.203127 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.203141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.203217 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.203408 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.203409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.203585 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.203653 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.203655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.204660 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.204661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.205216 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.205218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.205774 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.205775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.205841 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.206366 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.206371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.207661 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.209484 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.209485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.209485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.209659 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.209866 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.209866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.209867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.209921 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.210181 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.210182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.210221 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.210314 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.210394 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:13:53.210534 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.203160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.0.98.61:34745)
0924 00:13:53.203279 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.203281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.203290 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:53.203325 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.203326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.203336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.203402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.203746 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.204520 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.205358 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.205676 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.205757 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.207745 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.207746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.207747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.207754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.209394 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.209731 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.209733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.209733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.209796 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.209985 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.209986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.209986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.210120 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.210186 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.210360 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.210446 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:13:53.210929 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:53.211357 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.211166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58924)
0924 00:13:53.211304 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.211334 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:13:53.211724 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.211635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58938)
0924 00:13:53.211685 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.211709 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:13:53.211843 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
W20260924 00:13:53.240722 22733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:53.240952 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:53.240969 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:53.240989 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:53.242089 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:53.242129 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:53.242143 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:53.242156 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:53.242168 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:53.243031 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } attempt: 1
I20260924 00:13:53.243561 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.243404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58948)
0924 00:13:53.243476 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.243522 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 00:13:53.243988 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.243880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58964)
0924 00:13:53.243933 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.243963 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:13:53.244123 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
I20260924 00:13:53.244241 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34037,127.0.98.61:34745,127.0.98.60:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:53.244537 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:53.244771 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:53.249672 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:53.249682 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:53.249658 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:53.250101 22733 server_base.cc:1034] running on GCE node
I20260924 00:13:53.250365 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:53.250604 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:53.251801 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790208833251777 us; error 42 us; skew 500 ppm
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:53.254542 22733 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:13:53.255118 22733 webserver.cc:460] Webserver started at http://127.0.98.60:37983/ using document root <none> and password file <none>
I20260924 00:13:53.255411 22733 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:53.255484 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:53.255626 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:53.256721 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data/instance:
uuid: "1ad2e9d154434108a4ef5ed0ece047b4"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "8e964ede4c2a21b02e9cd03460c874b8"
server_key_iv: "c85d542f335e4c5e9c8587863b1aeac7"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.257090 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal/instance:
uuid: "1ad2e9d154434108a4ef5ed0ece047b4"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "8e964ede4c2a21b02e9cd03460c874b8"
server_key_iv: "c85d542f335e4c5e9c8587863b1aeac7"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.258550 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:53.259418 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:53.259598 22733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:53.259699 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "1ad2e9d154434108a4ef5ed0ece047b4"
format_stamp: "Formatted at 2026-09-24 00:13:53 on dist-test-slave-3q8k"
server_key: "8e964ede4c2a21b02e9cd03460c874b8"
server_key_iv: "c85d542f335e4c5e9c8587863b1aeac7"
server_key_version: "encryptionkey@0"
I20260924 00:13:53.259793 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:53.272761 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } attempt: 2
I20260924 00:13:53.273118 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:53.273284 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.273067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58978)
0924 00:13:53.273195 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.273228 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:13:53.273857 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.273744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:58992)
0924 00:13:53.273799 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.273827 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:13:53.273968 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41471: connect: Connection refused (error 111)
I20260924 00:13:53.275585 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:53.275746 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:53.280046 22733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41471
I20260924 00:13:53.280078 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41471 every 8 connection(s)
I20260924 00:13:53.280467 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:13:53.281457 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:53.283001 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:53.289789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22733
I20260924 00:13:53.289868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790208568194258-392-0/minicluster-data/master-2/wal/instance
I20260924 00:13:53.290072 392 external_mini_cluster.cc:1428] Setting key a4bc64f466000b9a04b6fa1e4ae25e92
I20260924 00:13:53.292739 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.283759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.98.62:34037)
0924 00:13:53.283863 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.283866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.284032 (+ 166us) server_negotiation.cc:407] Connection header received
0924 00:13:53.284565 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.284566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.284578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.284667 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.285338 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.285830 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.286831 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.287030 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.287110 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.289379 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.289381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.289381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.289389 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.291283 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.291740 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.291741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.291742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.291802 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.292071 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.292072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.292073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.292200 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.292274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.292559 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.292647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:13:53.292943 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.283681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.0.1:40314)
0924 00:13:53.283894 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.283940 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.284443 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.284710 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.284715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.284972 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.285232 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.285237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.285929 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.285931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.286734 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.286736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.287158 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.287159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.287337 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.287816 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.287825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.289268 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.291351 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.291386 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.291388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.291655 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.291878 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.291879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.291880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.291991 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.292280 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.292286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.292427 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.292538 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.292699 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260924 00:13:53.293483 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:53.300413 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.290462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.0.98.60:41471)
0924 00:13:53.290641 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.290646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.290696 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:13:53.290772 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.290775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.290794 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.290885 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.291617 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.292327 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.293264 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.293631 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.293717 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.296220 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.296226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.296227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.296233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.298077 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.298486 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.298489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.298489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.298543 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.298750 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.298752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.298752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.298939 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.299044 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.299189 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.300262 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260924 00:13:53.301692 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.293789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34745 (local address 127.0.0.1:34310)
0924 00:13:53.293825 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.293836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.293872 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.294028 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.294029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.294170 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.294216 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.294218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.294835 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.294837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.295652 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.295655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.296224 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.296225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.296298 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.296622 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.296626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.297820 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.300265 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.300266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.300267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.300591 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.300891 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.300892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.300893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.300959 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.301359 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.301360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.301410 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.301483 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.301596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 00:13:53.301859 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.293848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.0.98.61:34745)
0924 00:13:53.293892 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.293894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.293914 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:13:53.293946 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.293946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.293955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.294011 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.294309 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.294757 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.295767 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.296093 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.296142 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.297913 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.297915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.297915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.297923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.299543 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.300736 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.300737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.300737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.300811 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.301045 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.301046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.301046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.301280 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.301360 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.301693 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.301776 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:53.302211 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:53.309954 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.301450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.0.98.62:34037)
0924 00:13:53.301501 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.301503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.301511 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:53.301536 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.301537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.301548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.301609 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.302227 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.302954 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.303956 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.304266 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.304350 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.306860 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.306862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.306862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.306872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.308620 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309018 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.309019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.309019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.309074 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309300 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.309301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.309301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.309456 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.309673 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.309736 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.309826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:13:53.310590 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.302447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:59000)
0924 00:13:53.302523 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.302532 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.302566 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.302684 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.302685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.302825 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.302858 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.302859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.303772 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.303773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.304511 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.304514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.305035 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.305036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.305077 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.305527 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.305532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.307389 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.309307 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.309309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.309536 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.309743 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.309744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.309816 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.310126 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.310127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.310251 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.310349 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.310468 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:13:53.310917 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.302502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.0.98.60:41471)
0924 00:13:53.302541 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.302543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.302554 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:53.302623 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.302624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.302632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.302666 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.302914 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.303685 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.304605 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.304930 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.305015 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.307480 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.307481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.307481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.307487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.309222 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309608 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.309609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.309609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.309669 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.309894 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.309895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.309895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.310033 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.310099 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.310374 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.310552 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":19}
I20260924 00:13:53.312407 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: Bootstrap starting.
I20260924 00:13:53.313309 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:53.313604 22802 log.cc:824] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:53.314510 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: No bootstrap required, opened a new log
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:53.316500 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.316671 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad2e9d154434108a4ef5ed0ece047b4, State: Initialized, Role: FOLLOWER
I20260924 00:13:53.316758 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.317082 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.317148 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:53.317176 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:13:53.318943 22819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:53.319006 22819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:53.319272 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } attempt: 2
I20260924 00:13:53.319839 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.311051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.0.98.61:34745)
0924 00:13:53.311092 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.311094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.311101 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:53.311122 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.311123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.311148 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.311221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.311887 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.312428 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.313164 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.313439 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.313509 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.316359 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.316361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.316361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.316371 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.318087 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.318592 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.318593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.318594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.318660 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.318895 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.318896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.318897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.319045 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.319241 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.319654 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.319739 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:13:53.321048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:13:53.319398 22733 master_runner.cc:186] Error getting master registration for 127.0.98.62:34037: OK, instance_id { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" instance_seqno: 1790208832974055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:13:53.322242 22733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:13:53.322094 22625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1882f3b7f0894fe3909bd51acb63ce0f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40328
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:53.327064 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.319683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.0.98.60:41471)
0924 00:13:53.319813 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.319815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.319822 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:53.319840 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.319841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.319849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.320210 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.320241 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.321010 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.322000 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.322272 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.322369 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.324474 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.324476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.324476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.324483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.325887 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.326250 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.326251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.326251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.326318 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.326522 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.326523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.326523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.326662 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.326742 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.326901 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.326973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 00:13:53.327064 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.319494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:59010)
0924 00:13:53.319571 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.319581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.319618 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.319927 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.319928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.320095 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.320141 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.320143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.321144 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.321146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.321900 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.321902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.322376 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.322377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.322435 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.322902 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.322908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.324383 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.325991 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.325992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.325993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.326167 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.326395 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.326396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.326396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.326459 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.326734 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.326735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.326787 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.326876 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.326972 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 00:13:53.328461 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: Bootstrap starting.
I20260924 00:13:53.329190 22651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:53.329499 22651 log.cc:824] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:53.330364 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: No bootstrap required, opened a new log
I20260924 00:13:53.331504 22651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.331625 22651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1882f3b7f0894fe3909bd51acb63ce0f, State: Initialized, Role: FOLLOWER
I20260924 00:13:53.331723 22651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.332055 22651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:53.332015 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.332110 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:53.332216 22625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:13:53.332280 22625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:53.333166 22625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
W20260924 00:13:53.333894 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:53.333950 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:53.334161 22776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ad2e9d154434108a4ef5ed0ece047b4"
I20260924 00:13:53.334295 22776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:53.335167 22776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1882f3b7f0894fe3909bd51acb63ce0f in term 1.
I20260924 00:13:53.335592 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1882f3b7f0894fe3909bd51acb63ce0f, 1ad2e9d154434108a4ef5ed0ece047b4; no voters:
I20260924 00:13:53.335719 22822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:53.335865 22822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [term 1 LEADER]: Becoming Leader. State: Replica: 1882f3b7f0894fe3909bd51acb63ce0f, State: Running, Role: LEADER
I20260924 00:13:53.335952 22822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.336412 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1882f3b7f0894fe3909bd51acb63ce0f. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.336573 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:53.336381 22583 master_runner.cc:186] Error getting master registration for 127.0.98.61:34745: OK, instance_id { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" instance_seqno: 1790208833124557 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:13:53.338759 22583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:13:53.338837 22625 leader_election.cc:288] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [CANDIDATE]: Term 1 election: Requested vote from peers dcc9c533349445c1b638cc7ac49d3a55 (127.0.98.61:34745), 1ad2e9d154434108a4ef5ed0ece047b4 (127.0.98.60:41471)
I20260924 00:13:53.346953 22837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:53.347259 22837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:53.348845 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.340239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.60:41471)
0924 00:13:53.340318 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.340320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.340574 (+ 254us) server_negotiation.cc:407] Connection header received
0924 00:13:53.340780 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.340781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.340794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.340845 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.341717 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.342455 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.343566 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.343879 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.343949 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.345580 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.345582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.345582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.345590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.347429 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.347877 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.347878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.347879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.347957 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.348177 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.348178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.348178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.348333 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.348413 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.348633 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.348709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:13:53.354779 22776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [term 1 FOLLOWER]: Refusing update from remote peer 1882f3b7f0894fe3909bd51acb63ce0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:53.355116 22835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:53.356551 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1882f3b7f0894fe3909bd51acb63ce0f. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.356649 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:53.357003 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.340588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.0.98.61:34745)
0924 00:13:53.340960 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.340962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.349203 (+ 8241us) server_negotiation.cc:407] Connection header received
0924 00:13:53.349298 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.349299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.349311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.349340 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.350008 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.350648 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.351692 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.352012 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.352050 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.353665 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.353667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.353667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.353675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.355582 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.356041 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.356043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.356043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.356105 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.356323 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.356324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.356324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.356483 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.356563 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.356800 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.356888 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:13:53.359103 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.340194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.62:34037)
0924 00:13:53.340285 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.340287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.347558 (+ 7271us) server_negotiation.cc:407] Connection header received
0924 00:13:53.347725 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.347740 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.347755 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.347788 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.348830 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.349559 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.351972 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.352277 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.352321 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.355510 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.355512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.355512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.355520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.357462 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.358096 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.358097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.358097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.358152 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.358443 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.358444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.358444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.358577 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.358685 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.358921 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.359004 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:13:53.364169 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.364282 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:53.364409 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.364521 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:53.364907 22846 mvcc.cc:205] Tried to move back new op lower bound from 7332695381418455040 to 7332695381344870400. Current Snapshot: MvccSnapshot[applied={T|T < 7332695381418455040}]
I20260924 00:13:53.365096 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } attempt: 3
I20260924 00:13:53.366650 22837 catalog_manager.cc:1347] Generated new cluster ID: 09e53bb57c8c4433aa81dac5401a4bee
I20260924 00:13:53.366714 22837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:13:53 dist-test-slave-3q8k krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:53.376164 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.365365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41471 (local address 127.0.0.1:59028)
0924 00:13:53.365418 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:53.365429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:53.365470 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:53.365602 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:53.365603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:53.365793 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:53.365844 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.365846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.367016 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.367017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.367875 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:53.367878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.368670 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.368671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:53.368746 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.369825 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:53.369831 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:53.371611 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:53.374696 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.374697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.374698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.374928 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.375295 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:53.375296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:53.375296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:53.375372 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:53.375840 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:53.375841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:53.375888 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 00:13:53.375945 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:13:53.376055 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:13:53.377353 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:53.365407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.98.60:41471)
0924 00:13:53.365463 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.365465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.365475 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:13:53.365531 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.365532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.365543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.365597 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.365930 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.366855 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.368028 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:53.368555 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:53.370454 (+ 1899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:53.371728 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:53.371729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:53.371730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:53.371736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:53.374553 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.375063 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.375064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.375065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.375153 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:53.375459 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:53.375459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:53.375460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:53.375715 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:53.376593 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:53.376664 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:13:53.377164 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:13:53.379266 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: Bootstrap starting.
I20260924 00:13:53.380008 22726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:53.380352 22726 log.cc:824] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:53.386976 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: No bootstrap required, opened a new log
I20260924 00:13:53.389226 22726 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.389390 22726 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc9c533349445c1b638cc7ac49d3a55, State: Initialized, Role: FOLLOWER
I20260924 00:13:53.389482 22726 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } }
I20260924 00:13:53.389846 22726 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:53.390271 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.391172 22837 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:53.390435 22700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dcc9c533349445c1b638cc7ac49d3a55"
I20260924 00:13:53.390522 22699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:53.392015 22837 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:53.392917 22699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [term 1 FOLLOWER]: Refusing update from remote peer 1882f3b7f0894fe3909bd51acb63ce0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:53.393247 22822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:53.394250 22700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1882f3b7f0894fe3909bd51acb63ce0f in term 1.
I20260924 00:13:53.396170 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1882f3b7f0894fe3909bd51acb63ce0f. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.396052 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:53.396870 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:13:53.396814 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:53.398614 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:53.397446 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.398918 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:53.399528 22863 mvcc.cc:205] Tried to move back new op lower bound from 7332695381549314048 to 7332695381344870400. Current Snapshot: MvccSnapshot[applied={T|T < 7332695381418455040}]
I20260924 00:13:53.403136 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f3b7f0894fe3909bd51acb63ce0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } peers { permanent_uuid: "dcc9c533349445c1b638cc7ac49d3a55" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34745 } } peers { permanent_uuid: "1ad2e9d154434108a4ef5ed0ece047b4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41471 } } }
I20260924 00:13:53.403298 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:53.404664 22837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1882f3b7f0894fe3909bd51acb63ce0f: Generated new TSK 0
I20260924 00:13:53.404894 22837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:13:53.438053 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.436981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.98.60:41471)
0924 00:13:53.437029 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.437031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.437384 (+ 353us) server_negotiation.cc:407] Connection header received
0924 00:13:53.437496 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.437498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.437510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.437540 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.437985 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59032: BlockingRecv error: recv got EOF from 127.0.0.1:59032 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 00:13:53.439481 22654 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.438540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.98.62:34037)
0924 00:13:53.438581 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.438583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.438894 (+ 311us) server_negotiation.cc:407] Connection header received
0924 00:13:53.438993 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.438994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.439005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.439035 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.439419 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40342: BlockingRecv error: recv got EOF from 127.0.0.1:40342 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:13:53.440402 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.438773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.0.98.61:34745)
0924 00:13:53.438815 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.438816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.439861 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 00:13:53.439960 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.439961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.439972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.440002 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.440325 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34344: BlockingRecv error: recv got EOF from 127.0.0.1:34344 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:13:53.442791 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.441989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.98.61:34745)
0924 00:13:53.442025 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.442026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.442267 (+ 241us) server_negotiation.cc:407] Connection header received
0924 00:13:53.442378 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.442379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.442398 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.442430 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.442745 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34350: BlockingRecv error: recv got EOF from 127.0.0.1:34350 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 00:13:53.445834 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.445067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.98.61:34745)
0924 00:13:53.445100 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.445102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.445331 (+ 229us) server_negotiation.cc:407] Connection header received
0924 00:13:53.445430 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.445431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.445442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.445472 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.445787 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34358: BlockingRecv error: recv got EOF from 127.0.0.1:34358 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 00:13:53.449872 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.449092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.0.98.61:34745)
0924 00:13:53.449127 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.449129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.449358 (+ 229us) server_negotiation.cc:407] Connection header received
0924 00:13:53.449456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.449457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.449468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.449497 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.449824 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34368: BlockingRecv error: recv got EOF from 127.0.0.1:34368 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 00:13:53.456198 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.455384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.98.61:34745)
0924 00:13:53.455421 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.455423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.455652 (+ 229us) server_negotiation.cc:407] Connection header received
0924 00:13:53.455758 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.455759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.455769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.455799 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.456133 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34380: BlockingRecv error: recv got EOF from 127.0.0.1:34380 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 00:13:53.466451 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.465550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.0.98.61:34745)
0924 00:13:53.465591 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.465593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.465841 (+ 248us) server_negotiation.cc:407] Connection header received
0924 00:13:53.465946 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.465947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.465960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.465991 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.466380 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34386: BlockingRecv error: recv got EOF from 127.0.0.1:34386 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 00:13:53.484917 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.483974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.0.98.61:34745)
0924 00:13:53.484020 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.484022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.484302 (+ 280us) server_negotiation.cc:407] Connection header received
0924 00:13:53.484411 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.484412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.484424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.484455 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.484856 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34390: BlockingRecv error: recv got EOF from 127.0.0.1:34390 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 00:13:53.519447 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.518613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.0.98.61:34745)
0924 00:13:53.518663 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.518666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.518801 (+ 135us) server_negotiation.cc:407] Connection header received
0924 00:13:53.518900 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.518901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.518913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.518988 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.519372 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 00:13:53.585454 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.584735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.0.98.61:34745)
0924 00:13:53.584808 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.584811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.584903 (+ 92us) server_negotiation.cc:407] Connection header received
0924 00:13:53.584973 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.584974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.584984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.585029 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.585382 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34412: BlockingRecv error: recv got EOF from 127.0.0.1:34412 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 00:13:53.715803 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.714846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.0.98.61:34745)
0924 00:13:53.714948 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.714950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.715047 (+ 97us) server_negotiation.cc:407] Connection header received
0924 00:13:53.715083 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.715084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.715097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.715365 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.715748 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34414: BlockingRecv error: recv got EOF from 127.0.0.1:34414 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 00:13:53.973692 22731 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:53.973016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.0.98.61:34745)
0924 00:13:53.973097 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:53.973099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:53.973109 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:13:53.973186 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:53.973187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:53.973198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:53.973255 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:53.973623 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34418: BlockingRecv error: recv got EOF from 127.0.0.1:34418 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:13:54.321506 22819 catalog_manager.cc:1259] Loaded cluster ID: 09e53bb57c8c4433aa81dac5401a4bee
I20260924 00:13:54.321609 22819 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: loading cluster ID for follower catalog manager: success
I20260924 00:13:54.322438 22819 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: acquiring CA information for follower catalog manager: success
I20260924 00:13:54.322746 22819 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ad2e9d154434108a4ef5ed0ece047b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:13:54.399103 22866 catalog_manager.cc:1259] Loaded cluster ID: 09e53bb57c8c4433aa81dac5401a4bee
I20260924 00:13:54.399207 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: loading cluster ID for follower catalog manager: success
I20260924 00:13:54.400028 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: acquiring CA information for follower catalog manager: success
I20260924 00:13:54.400346 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcc9c533349445c1b638cc7ac49d3a55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:13:54.489432 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:54.486861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.0.98.61:34745)
0924 00:13:54.486970 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:54.486972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:54.486981 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:54.487053 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:54.487054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:54.487101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:54.487173 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:54.487365 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.487860 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.488582 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.488777 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.488832 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:54.488999 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:54.489104 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:54.489254 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:54.489270 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 00:13:54.489290 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260924 00:13:54.493860 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:54.491385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.0.98.60:41471)
0924 00:13:54.491499 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:54.491500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:54.491509 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:54.491587 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:54.491587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:54.491624 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:54.491687 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:54.491786 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.492281 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.493019 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.493220 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.493275 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:54.493453 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:54.493558 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:54.493618 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:54.493706 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:13:54.493724 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 00:13:54.498682 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:54.495889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.60:41471)
0924 00:13:54.496004 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:54.496007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:54.496016 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:54.496105 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:54.496106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:54.496119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:54.496156 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:54.496343 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.497100 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.497990 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.498192 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.498267 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:54.498424 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:54.498457 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:54.498484 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:54.498580 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 00:13:54.498600 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 00:13:54.499014 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:54.495903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.0.98.62:34037)
0924 00:13:54.496068 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:54.496070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:54.496078 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:54.496123 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:54.496124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:54.496187 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:54.496278 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:54.496437 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.497046 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.498043 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.498354 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.498394 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:54.498633 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:54.498730 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:54.498806 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:54.498911 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:13:54.498933 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":58,"threads_started":1}
I20260924 00:13:54.499255 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:54.496155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.61:34745)
0924 00:13:54.496269 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:54.496272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:54.496433 (+ 161us) server_negotiation.cc:407] Connection header received
0924 00:13:54.496537 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:54.496538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:54.496548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:54.496604 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:54.496739 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.497428 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.498397 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:54.498694 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:54.498753 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:54.498957 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:54.498991 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:54.499021 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:54.499136 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 00:13:54.499164 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 00:13:54.501183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22583
I20260924 00:13:54.506449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22658
I20260924 00:13:54.511564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22733
2026-09-24T00:13:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2830 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:13:54 dist-test-slave-3q8k krb5kdc[22949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:13:54 dist-test-slave-3q8k krb5kdc[22949](info): set up 2 sockets
Sep 24 00:13:54 dist-test-slave-3q8k krb5kdc[22949](info): commencing operation
krb5kdc: starting...
W20260924 00:13:55.555649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/test-user.keytab.
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:13:55.682375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:13:55.771528 22962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:55.771909 22962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:55.771948 22962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:55.771971 22962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:55.773167 22962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:55.773211 22962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:55.773226 22962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:55.773238 22962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:55.773250 22962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:55.774797 22962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:55.775013 22962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:55.775198 22962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:55.780854 22967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:55.780925 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:55.780864 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:55.781257 22962 server_base.cc:1034] running on GCE node
I20260924 00:13:55.781556 22962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:55.781865 22962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:55.783042 22962 hybrid_clock.cc:648] HybridClock initialized: now 1790208835783033 us; error 55 us; skew 500 ppm
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:55.785830 22962 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:13:55.786278 22962 webserver.cc:460] Webserver started at http://127.0.98.62:38243/ using document root <none> and password file <none>
I20260924 00:13:55.786513 22962 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:55.786583 22962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:55.786744 22962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:55.787904 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data/instance:
uuid: "ec782d36365842ac8636a2e309307fb2"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "7bdbdb42ec24c1e01e71a5b5f406ec43"
server_key_iv: "8779e7c8417113e44cad4cba2ce188b1"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.788266 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec782d36365842ac8636a2e309307fb2"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "7bdbdb42ec24c1e01e71a5b5f406ec43"
server_key_iv: "8779e7c8417113e44cad4cba2ce188b1"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.789575 22962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:13:55.790328 22977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:55.790496 22962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:55.790602 22962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal
uuid: "ec782d36365842ac8636a2e309307fb2"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "7bdbdb42ec24c1e01e71a5b5f406ec43"
server_key_iv: "8779e7c8417113e44cad4cba2ce188b1"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.790702 22962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:55.816481 22962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:55.819087 22962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:55.819298 22962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:55.823125 22962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42237
I20260924 00:13:55.823191 23029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42237 every 8 connection(s)
I20260924 00:13:55.823613 22962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:13:55.824602 23030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:55.826246 23030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:55.828202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 22962
I20260924 00:13:55.828312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-0/wal/instance
I20260924 00:13:55.828512 392 external_mini_cluster.cc:1428] Setting key 51f1f168c60eebca345b8f9fde2cc669
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 00:13:55.837402 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.826963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42237 (local address 127.0.0.1:58614)
0924 00:13:55.827168 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.827248 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:55.827723 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:55.828072 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:55.828075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:55.828522 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:55.828789 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.828796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.829920 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.829921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.830608 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.830610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.831408 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.831409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.831474 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.832109 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:55.832117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:55.833649 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:55.835855 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.835862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.835863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.836058 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.836344 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.836345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.836345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.836421 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.836897 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:55.836911 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:55.836970 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:13:55.837048 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:13:55.837188 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260924 00:13:55.837409 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.827065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.0.98.62:42237)
0924 00:13:55.827251 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:55.827261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:55.827321 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:13:55.827863 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:55.827865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:55.827883 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:55.827988 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:55.828909 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.829788 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.830713 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.831291 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.831403 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.833748 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:55.833757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:55.833758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:55.833766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:55.835750 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.836148 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.836152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.836153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.836265 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.836490 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.836492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.836492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.836798 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:55.836908 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:55.837107 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 00:13:55.837246 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260924 00:13:55.838250 23030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:55.838598 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.838478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42712)
0924 00:13:55.838528 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.838575 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 00:13:55.838794 22978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
I20260924 00:13:55.839205 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.839126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42718)
0924 00:13:55.839165 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.839189 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 00:13:55.839340 23030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43737: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:13:55.854813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43737
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:55.888589 23030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } attempt: 1
I20260924 00:13:55.889158 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.888999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42730)
0924 00:13:55.889071 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.889122 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 00:13:55.889567 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.889467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42738)
0924 00:13:55.889515 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.889546 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:13:55.889668 23030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43737: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
W20260924 00:13:55.939934 23037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:55.940155 23037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:55.940176 23037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:55.940196 23037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:55.941440 23037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:55.941488 23037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:55.941502 23037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:55.941515 23037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:55.941532 23037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:55.943094 23037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:55.943387 23037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:55.943626 23037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:55.948608 23045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:55.948609 23042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:55.948649 23043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:55.948799 23037 server_base.cc:1034] running on GCE node
I20260924 00:13:55.949036 23037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:55.949369 23037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:55.950522 23037 hybrid_clock.cc:648] HybridClock initialized: now 1790208835950514 us; error 30 us; skew 500 ppm
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:55.953145 23037 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:13:55.953581 23037 webserver.cc:460] Webserver started at http://127.0.98.61:43579/ using document root <none> and password file <none>
I20260924 00:13:55.953851 23037 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:55.953924 23037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:55.954070 23037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:55.955147 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data/instance:
uuid: "9a27d782c166411584909097660e7f84"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "3d3babacb3a3adfea2c59e3856c926b7"
server_key_iv: "bed50eeaf7eb005f4e510124c28f64ef"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.955585 23037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal/instance:
uuid: "9a27d782c166411584909097660e7f84"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "3d3babacb3a3adfea2c59e3856c926b7"
server_key_iv: "bed50eeaf7eb005f4e510124c28f64ef"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.956928 23037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:55.957660 23052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:55.957845 23037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:13:55.957818 23030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } attempt: 2
I20260924 00:13:55.957948 23037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal
uuid: "9a27d782c166411584909097660e7f84"
format_stamp: "Formatted at 2026-09-24 00:13:55 on dist-test-slave-3q8k"
server_key: "3d3babacb3a3adfea2c59e3856c926b7"
server_key_iv: "bed50eeaf7eb005f4e510124c28f64ef"
server_key_version: "encryptionkey@0"
I20260924 00:13:55.958031 23037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:55.958277 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.958142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42744)
0924 00:13:55.958215 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.958244 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:13:55.958665 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.958555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42760)
0924 00:13:55.958610 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.958641 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 00:13:55.958776 23030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43737: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43737: connect: Connection refused (error 111)
I20260924 00:13:55.970147 23037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:55.972618 23037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:55.972787 23037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:55.976825 23037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43737
I20260924 00:13:55.976920 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43737 every 8 connection(s)
I20260924 00:13:55.977229 23037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:13:55.978200 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:55.979771 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:55.980026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 23037
I20260924 00:13:55.980175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-1/wal/instance
I20260924 00:13:55.980373 392 external_mini_cluster.cc:1428] Setting key 17118186998987d488efb4127ce30c9d
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 00:13:55.990319 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.980547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.62:42237)
0924 00:13:55.980608 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:55.980611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:55.980878 (+ 267us) server_negotiation.cc:407] Connection header received
0924 00:13:55.981727 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:55.981728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:55.981742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:55.981867 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:55.982510 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.983328 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.984432 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.984781 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.984872 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.987554 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:55.987556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:55.987556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:55.987576 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:55.989071 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.989428 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.989428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.989429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.989478 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.989750 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.989751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.989752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.989889 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:55.989947 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:55.990190 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 00:13:55.990252 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:13:55.990494 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.980529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42237 (local address 127.0.0.1:58616)
0924 00:13:55.980741 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.980787 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:55.981597 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:55.981945 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:55.981950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:55.982226 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:55.982416 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.982421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.983424 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.983425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.984326 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.984329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.984901 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.984902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.985020 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.985778 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:55.985787 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:55.987456 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:55.989145 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.989152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.989153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.989359 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.989537 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.989537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.989538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.989687 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.989959 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:55.989964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:55.990076 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 00:13:55.990169 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:13:55.990286 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260924 00:13:55.991143 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 00:13:55 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:55.999634 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.991485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42774)
0924 00:13:55.991694 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:55.991705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:55.991754 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:55.991912 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:55.991913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:55.992026 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:55.992076 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.992078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.993375 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.993377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.994270 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:55.994273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.994769 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.994770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:55.994873 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.995321 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:55.995327 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:55.996717 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:55.998524 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.998524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.998525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.998699 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.998913 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:55.998914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:55.998915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:55.998968 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:55.999301 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:55.999302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:55.999342 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 00:13:55.999413 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:13:55.999513 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:13:55.999718 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:55.991560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.61:43737)
0924 00:13:55.991704 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:55.991708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:55.991762 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:13:55.991813 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:55.991815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:55.991831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:55.991912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:55.992184 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.993300 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.994348 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:55.994699 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:55.994754 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:55.996815 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:55.996824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:55.996825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:55.996833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:55.998396 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.998789 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.998793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.998793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.998849 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:55.999048 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:55.999049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:55.999050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:55.999191 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:55.999323 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:55.999443 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 00:13:55.999581 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 00:13:56.000262 23105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:56.000640 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.000498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41706)
0924 00:13:56.000564 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.000616 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 00:13:56.000864 23055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
I20260924 00:13:56.001276 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.001191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41720)
0924 00:13:56.001235 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.001259 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 00:13:56.001400 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33005: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:13:56.008059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskElJ0hL/build/release/bin/kudu
/tmp/dist-test-taskElJ0hL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33005
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:13:56.049938 23030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } attempt: 3
I20260924 00:13:56.050627 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } attempt: 1
I20260924 00:13:56.051055 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.050917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41724)
0924 00:13:56.050983 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.051023 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 00:13:56.052125 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.051995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41726)
0924 00:13:56.052052 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.052094 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 00:13:56.052227 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33005: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:56.057608 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.050463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.0.98.61:43737)
0924 00:13:56.050533 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.050536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.050546 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:13:56.050628 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.050629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.050641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.050735 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.051119 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.051681 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.052636 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.052848 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.052936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.054978 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.054980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.054980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.054988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.056443 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.056812 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.056813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.056814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.056874 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.057072 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.057073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.057073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.057192 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.057270 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.057425 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.057506 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:13:56.057608 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.050387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42784)
0924 00:13:56.050448 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.050465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.050544 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.050746 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.050746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.050947 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.051023 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.051026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.051782 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.051783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.052525 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.052528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.052939 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.052939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.053022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.053524 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.053529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.054888 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.056558 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.056559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.056560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.056736 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.056943 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.056944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.056944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.057008 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.057262 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.057263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.057313 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.057400 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.057506 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:13:56.057945 23030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:56.058306 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.058206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41742)
0924 00:13:56.058255 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.058282 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:13:56.058693 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.058632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41758)
0924 00:13:56.058658 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.058680 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 00:13:56.058799 23030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33005: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
I20260924 00:13:56.089946 23030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } attempt: 1
I20260924 00:13:56.090519 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.090347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41768)
0924 00:13:56.090430 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.090479 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 00:13:56.090976 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.090865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41770)
0924 00:13:56.090914 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.090951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:13:56.091076 23030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33005: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
W20260924 00:13:56.092597 23112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:13:56.092846 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:13:56.092867 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:13:56.092927 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:13:56.094141 23112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:13:56.094182 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:13:56.094197 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:13:56.094209 23112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:13:56.094223 23112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:13:56.095796 23112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42237,127.0.98.61:43737,127.0.98.60:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:13:56.096055 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:13:56.096299 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:13:56.101562 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:13:56.101583 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:56.101632 23112 server_base.cc:1034] running on GCE node
W20260924 00:13:56.101616 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:13:56.102110 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:13:56.102444 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:13:56.103803 23112 hybrid_clock.cc:648] HybridClock initialized: now 1790208836103758 us; error 59 us; skew 500 ppm
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:13:56.106523 23112 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:13:56.107152 23112 webserver.cc:460] Webserver started at http://127.0.98.60:34599/ using document root <none> and password file <none>
I20260924 00:13:56.107409 23112 fs_manager.cc:359] Metadata directory not provided
I20260924 00:13:56.107481 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:13:56.107636 23112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:13:56.108738 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data/instance:
uuid: "a3bf42e4ba6c48ee90575ce3a66421f6"
format_stamp: "Formatted at 2026-09-24 00:13:56 on dist-test-slave-3q8k"
server_key: "cd6cae79be6ed9ee17c1c453563ceb65"
server_key_iv: "915a7f3341be65649595c1c9a28fe420"
server_key_version: "encryptionkey@0"
I20260924 00:13:56.109107 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal/instance:
uuid: "a3bf42e4ba6c48ee90575ce3a66421f6"
format_stamp: "Formatted at 2026-09-24 00:13:56 on dist-test-slave-3q8k"
server_key: "cd6cae79be6ed9ee17c1c453563ceb65"
server_key_iv: "915a7f3341be65649595c1c9a28fe420"
server_key_version: "encryptionkey@0"
I20260924 00:13:56.110451 23112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:13:56.111322 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:13:56.111505 23112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:13:56.111604 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal
uuid: "a3bf42e4ba6c48ee90575ce3a66421f6"
format_stamp: "Formatted at 2026-09-24 00:13:56 on dist-test-slave-3q8k"
server_key: "cd6cae79be6ed9ee17c1c453563ceb65"
server_key_iv: "915a7f3341be65649595c1c9a28fe420"
server_key_version: "encryptionkey@0"
I20260924 00:13:56.111711 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:13:56.118975 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:13:56.120601 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } attempt: 2
I20260924 00:13:56.121032 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.120884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41780)
0924 00:13:56.120967 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.121004 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:13:56.121407 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.121305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41788)
0924 00:13:56.121356 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.121382 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:13:56.121500 23105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33005: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33005: connect: Connection refused (error 111)
I20260924 00:13:56.121883 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:13:56.122047 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:13:56.126322 23112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33005
I20260924 00:13:56.126354 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33005 every 8 connection(s)
I20260924 00:13:56.126731 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:13:56.127812 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:13:56.129384 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:56.133140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskElJ0hL/build/release/bin/kudu as pid 23112
I20260924 00:13:56.133313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskElJ0hL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790208568194258-392-0/minicluster-data/master-2/wal/instance
I20260924 00:13:56.133496 392 external_mini_cluster.cc:1428] Setting key e74684539444f3c43debee797c16c14f
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:56.138934 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.130113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.98.62:42237)
0924 00:13:56.130243 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.130246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.130401 (+ 155us) server_negotiation.cc:407] Connection header received
0924 00:13:56.130952 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.130954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.130967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.131031 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.131714 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.132232 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.133183 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.133472 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.133555 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.135865 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.135866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.135867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.135874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.137356 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.137859 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.137860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.137861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.137913 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.138255 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.138255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.138256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.138358 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.138410 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.138754 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.138839 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 00:13:56.139036 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.130097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42237 (local address 127.0.0.1:58632)
0924 00:13:56.130276 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.130319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.130821 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.131113 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.131116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.131419 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.131614 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.131618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.132322 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.132323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.133092 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.133093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.133584 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.133585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.133725 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.134297 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.134306 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.135787 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.137454 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.137460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.137461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.137780 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.137981 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.137982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.137983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.138129 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.138426 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.138432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.138589 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.138717 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.138849 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:56.139683 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:13:56.143069 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.133928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.0.98.60:33005)
0924 00:13:56.134084 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.134089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.134232 (+ 143us) server_negotiation.cc:407] Connection header received
0924 00:13:56.134295 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.134298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.134326 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.134406 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.135114 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.135978 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.136769 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.137064 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.137142 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.139994 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.140001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.140002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.140009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.141696 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.142086 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.142089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.142089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.142147 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.142388 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.142389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.142389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.142592 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.142775 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.142919 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.142981 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:56.147410 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.139945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43737 (local address 127.0.0.1:42790)
0924 00:13:56.139978 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.139987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.140014 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.140173 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.140174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.140339 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.140398 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.140400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.141353 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.141353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.141903 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.141906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.142853 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.142855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.142934 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.143403 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.143407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.144728 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.146375 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.146376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.146376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.146545 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.146757 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.146758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.146759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.146818 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.147067 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.147068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.147110 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.147193 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.147316 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 00:13:56.147460 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.140004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.98.61:43737)
0924 00:13:56.140051 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.140053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.140060 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:56.140083 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.140084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.140094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.140174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.140474 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.141264 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.141991 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.142765 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.142831 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.144832 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.144834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.144834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.144842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.146255 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.146617 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.146619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.146619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.146676 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.146882 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.146883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.146883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.146999 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.147074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.147290 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.147366 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:13:56.147706 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:13:56.151306 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.144037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.0.98.62:42237)
0924 00:13:56.144079 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.144081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.144133 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:13:56.144183 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.144184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.144194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.144261 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.144728 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.145226 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.146062 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.146267 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.146351 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.148658 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.148660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.148660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.148667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.150042 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.150451 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.150453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.150453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.150509 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.150742 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.150743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.150743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.150875 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.151050 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.151101 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.151177 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:56.154708 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.148046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.98.60:33005)
0924 00:13:56.148097 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.148099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.148106 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:56.148123 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.148124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.148133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.148187 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.148456 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.148978 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.149803 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.150023 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.150106 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.151916 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.151917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.151917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.151924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.153542 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.153885 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.153886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.153886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.153942 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.154149 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.154149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.154150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.154247 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.154297 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.154482 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.154603 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":35}
I20260924 00:13:56.154707 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.147980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41814)
0924 00:13:56.148019 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.148028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.148077 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.148191 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.148191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.148346 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.148376 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.148377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.149061 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.149062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.149705 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.149707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.150106 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.150107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.150155 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.150544 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.150549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.151828 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.153618 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.153619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.153619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.153774 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.154004 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.154004 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.154005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.154082 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.154310 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.154311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.154386 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.154466 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.154563 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 00:13:56.156307 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: Bootstrap starting.
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:13:56.157366 23180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:56.157691 23180 log.cc:824] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:56.158460 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: No bootstrap required, opened a new log
I20260924 00:13:56.159976 23180 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.160123 23180 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bf42e4ba6c48ee90575ce3a66421f6, State: Initialized, Role: FOLLOWER
I20260924 00:13:56.160212 23180 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.160332 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.152634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.61:43737)
0924 00:13:56.152697 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.152699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.152708 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:56.152730 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.152731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.152742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.152798 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.153330 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.153999 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.154946 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.155168 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.155287 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.157664 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.157665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.157665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.157672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.159145 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.159561 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.159562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.159562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.159617 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.159834 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.159835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.159836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.159954 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.160113 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.160155 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.160232 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:13:56.160594 23180 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:56.160530 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.160674 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.161962 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 00:13:56.162642 23197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:56.162695 23197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:56.162889 23004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ec782d36365842ac8636a2e309307fb2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58648
I20260924 00:13:56.162919 23112 master_runner.cc:186] Error getting master registration for 127.0.98.62:42237: OK, instance_id { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" instance_seqno: 1790208835822091 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:13:56.165807 23112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:13:56.166231 23030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } attempt: 2
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:56.174114 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.166530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.98.60:33005)
0924 00:13:56.166573 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.166575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.166627 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:13:56.166692 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.166692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.166702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.166792 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.167107 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.167849 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.168832 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.169119 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.169181 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.171467 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.171469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.171469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.171476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.172909 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.173293 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.173295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.173295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.173354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.173561 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.173562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.173562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.173698 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.173778 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.173937 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.174012 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:13:56.174114 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.166475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41826)
0924 00:13:56.166581 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.166592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.166640 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.166801 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.166802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.166980 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.167029 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.167031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.167932 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.167934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.168739 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.168741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.169192 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.169193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.169258 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.169755 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.169759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.171385 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.173016 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.173017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.173018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.173217 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.173428 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.173429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.173429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.173494 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.173780 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.173781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.173827 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.173913 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.174012 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 00:13:56.175962 23030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: Bootstrap starting.
I20260924 00:13:56.176776 23030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:56.177073 23030 log.cc:824] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:56.177954 23030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: No bootstrap required, opened a new log
I20260924 00:13:56.179346 23030 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.179493 23030 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec782d36365842ac8636a2e309307fb2, State: Initialized, Role: FOLLOWER
I20260924 00:13:56.179579 23030 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.179883 23030 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:56.179941 23004 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:13:56.179873 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.180042 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.180073 23004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:56.180936 23004 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.181442 23004 leader_election.cc:288] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a27d782c166411584909097660e7f84 (127.0.98.61:43737), a3bf42e4ba6c48ee90575ce3a66421f6 (127.0.98.60:33005)
I20260924 00:13:56.181703 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec782d36365842ac8636a2e309307fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6"
I20260924 00:13:56.181851 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:13:56.182467 23211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:56.182578 23211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:56.182742 23154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec782d36365842ac8636a2e309307fb2 in term 1.
I20260924 00:13:56.182958 22980 leader_election.cc:302] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3bf42e4ba6c48ee90575ce3a66421f6, ec782d36365842ac8636a2e309307fb2; no voters:
I20260924 00:13:56.183185 23200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:13:56.183511 23200 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [term 1 LEADER]: Becoming Leader. State: Replica: ec782d36365842ac8636a2e309307fb2, State: Running, Role: LEADER
I20260924 00:13:56.183604 23200 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.184109 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec782d36365842ac8636a2e309307fb2. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.184180 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:56.184981 22962 master_runner.cc:186] Error getting master registration for 127.0.98.61:43737: OK, instance_id { permanent_uuid: "9a27d782c166411584909097660e7f84" instance_seqno: 1790208835975606 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:13:56.187374 22962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:13:56.188063 23222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:13:56.188308 23222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:13:56.192883 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.183029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.98.60:33005)
0924 00:13:56.183066 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.183068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.183627 (+ 559us) server_negotiation.cc:407] Connection header received
0924 00:13:56.183780 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.183781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.183791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.183833 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.184980 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.185687 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.187205 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.187532 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.187572 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.189529 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.189541 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.189542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.189549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.191209 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.191822 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.191823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.191823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.191879 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.192171 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.192172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.192172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.192316 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.192399 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.192706 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.192787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:13:56.195060 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.183310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.98.62:42237)
0924 00:13:56.183378 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.183380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.183945 (+ 565us) server_negotiation.cc:407] Connection header received
0924 00:13:56.184046 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.184047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.184058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.184088 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.184805 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.185512 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.186555 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.186833 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.186876 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.190921 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.190922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.190923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.190930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.192869 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.193985 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.193987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.193987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.194058 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.194355 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.194356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.194356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.194496 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.194579 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.194891 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.194968 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:13:56.196477 23154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [term 1 FOLLOWER]: Refusing update from remote peer ec782d36365842ac8636a2e309307fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:56.196838 23219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:13:56.197837 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.183615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.0.98.61:43737)
0924 00:13:56.183660 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.183663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.189190 (+ 5527us) server_negotiation.cc:407] Connection header received
0924 00:13:56.189291 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.189292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.189304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.189334 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.189985 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.190778 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.191890 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.192222 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.192267 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.194137 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.194138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.194139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.194146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.196000 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.196529 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.196531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.196531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.196588 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.197212 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.197213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.197214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.197351 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.197437 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.197651 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.197733 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:13:56.198748 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec782d36365842ac8636a2e309307fb2. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.198841 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.205420 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.205518 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:56.205564 23222 catalog_manager.cc:1347] Generated new cluster ID: e2133d2829df427e8419317697af2a0f
I20260924 00:13:56.205638 23222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:13:56.206149 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.206233 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.206574 23223 mvcc.cc:205] Tried to move back new op lower bound from 7332695393045880832 to 7332695393009192960. Current Snapshot: MvccSnapshot[applied={T|T < 7332695393045880832 or (T in {7332695393045880832})}]
I20260924 00:13:56.212620 23105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } attempt: 3
I20260924 00:13:56.214150 23224 mvcc.cc:205] Tried to move back new op lower bound from 7332695393045880832 to 7332695393009192960. Current Snapshot: MvccSnapshot[applied={T|T < 7332695393045880832 or (T in {7332695393045880832})}]
Sep 24 00:13:56 dist-test-slave-3q8k krb5kdc[22949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790208835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:13:56.221549 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.212886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33005 (local address 127.0.0.1:41842)
0924 00:13:56.212976 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 00:13:56.212989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:13:56.213035 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:13:56.213575 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:13:56.213576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:13:56.213766 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:13:56.213816 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.213818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.214746 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.214747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.215636 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:13:56.215639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.216206 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.216207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:13:56.216278 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.216764 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:13:56.216769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:13:56.218358 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:13:56.220308 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.220309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.220310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.220539 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.220798 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:13:56.220799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:13:56.220799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:13:56.220882 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:13:56.221217 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:13:56.221218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:13:56.221277 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:13:56.221338 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 00:13:56.221442 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 00:13:56.222126 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:56.213120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.98.60:33005)
0924 00:13:56.213180 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.213183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.213194 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:13:56.213218 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.213219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.213233 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.213302 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.213903 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.214662 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.215752 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:56.216069 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:56.216131 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:56.218472 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:13:56.218474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:13:56.218474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:13:56.218481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:13:56.220192 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.220638 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.220639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.220640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.220713 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:13:56.220963 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:13:56.220963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:13:56.220964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:13:56.221113 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:13:56.221173 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:56.221913 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 00:13:56.222007 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:13:56.224193 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: Bootstrap starting.
I20260924 00:13:56.224998 23105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: Neither blocks nor log segments found. Creating new log.
I20260924 00:13:56.225322 23105 log.cc:824] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: Log is configured to *not* fsync() on all Append() calls
I20260924 00:13:56.228055 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: No bootstrap required, opened a new log
I20260924 00:13:56.230099 23105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.230237 23105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a27d782c166411584909097660e7f84, State: Initialized, Role: FOLLOWER
I20260924 00:13:56.230312 23105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } }
I20260924 00:13:56.230646 23105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:13:56.231331 23078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:13:56.232260 23078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [term 1 FOLLOWER]: Refusing update from remote peer ec782d36365842ac8636a2e309307fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:13:56.232441 23079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec782d36365842ac8636a2e309307fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a27d782c166411584909097660e7f84"
I20260924 00:13:56.233175 23079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec782d36365842ac8636a2e309307fb2 in term 1.
I20260924 00:13:56.233634 23227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:13:56.238670 23245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:13:56.238863 23245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:13:56.239001 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.239123 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.239212 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec782d36365842ac8636a2e309307fb2. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.239333 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.239404 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.239497 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:13:56.239877 23235 mvcc.cc:205] Tried to move back new op lower bound from 7332695393045880832 to 7332695393009192960. Current Snapshot: MvccSnapshot[applied={T|T < 7332695393045880832}]
I20260924 00:13:56.241216 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec782d36365842ac8636a2e309307fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec782d36365842ac8636a2e309307fb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42237 } } peers { permanent_uuid: "9a27d782c166411584909097660e7f84" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43737 } } peers { permanent_uuid: "a3bf42e4ba6c48ee90575ce3a66421f6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33005 } } }
I20260924 00:13:56.241320 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:13:56.242043 23222 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:13:56.242847 23222 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:13:56.258620 23222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec782d36365842ac8636a2e309307fb2: Generated new TSK 0
I20260924 00:13:56.258836 23222 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:13:56.315132 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.307832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.0.98.60:33005)
0924 00:13:56.307873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.307875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.314108 (+ 6233us) server_negotiation.cc:407] Connection header received
0924 00:13:56.314409 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.314411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.314425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.314461 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.315059 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41848: BlockingRecv error: recv got EOF from 127.0.0.1:41848 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:13:56.315658 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.307561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.0.98.62:42237)
0924 00:13:56.307612 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.307614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.314211 (+ 6597us) server_negotiation.cc:407] Connection header received
0924 00:13:56.314539 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.314540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.314553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.314585 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.315588 (+ 1003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58670: BlockingRecv error: recv got EOF from 127.0.0.1:58670 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 00:13:56.315713 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.308313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.0.98.61:43737)
0924 00:13:56.308371 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.308373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.314389 (+ 6016us) server_negotiation.cc:407] Connection header received
0924 00:13:56.314488 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.314490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.314503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.314533 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.315631 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42802: BlockingRecv error: recv got EOF from 127.0.0.1:42802 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 00:13:56.332038 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.327562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.0.98.61:43737)
0924 00:13:56.327615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.327618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.331306 (+ 3688us) server_negotiation.cc:407] Connection header received
0924 00:13:56.331450 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.331451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.331476 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.331509 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.331975 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42814: BlockingRecv error: recv got EOF from 127.0.0.1:42814 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 00:13:56.354990 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.341480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.98.61:43737)
0924 00:13:56.341527 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.341530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.354307 (+ 12777us) server_negotiation.cc:407] Connection header received
0924 00:13:56.354435 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.354436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.354450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.354482 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.354920 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42818: BlockingRecv error: recv got EOF from 127.0.0.1:42818 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:13:56.373890 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.361787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.0.98.61:43737)
0924 00:13:56.361847 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.361849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.373085 (+ 11236us) server_negotiation.cc:407] Connection header received
0924 00:13:56.373257 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.373258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.373274 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.373320 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.373812 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42834: BlockingRecv error: recv got EOF from 127.0.0.1:42834 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 00:13:56.393803 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.387588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.0.98.61:43737)
0924 00:13:56.387642 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.387644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.393094 (+ 5450us) server_negotiation.cc:407] Connection header received
0924 00:13:56.393227 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.393228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.393243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.393274 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.393737 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42842: BlockingRecv error: recv got EOF from 127.0.0.1:42842 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 00:13:56.420163 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.415603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.0.98.61:43737)
0924 00:13:56.415655 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.415657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.419531 (+ 3874us) server_negotiation.cc:407] Connection header received
0924 00:13:56.419604 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.419606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.419620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.419700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.420104 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42858: BlockingRecv error: recv got EOF from 127.0.0.1:42858 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 00:13:56.438304 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.437574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.0.98.61:43737)
0924 00:13:56.437632 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.437635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.437754 (+ 119us) server_negotiation.cc:407] Connection header received
0924 00:13:56.437790 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.437791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.437804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.437885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.438241 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42860: BlockingRecv error: recv got EOF from 127.0.0.1:42860 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 00:13:56.472527 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.471607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.98.61:43737)
0924 00:13:56.471797 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.471800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.471896 (+ 96us) server_negotiation.cc:407] Connection header received
0924 00:13:56.471993 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.471994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.472005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.472081 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.472456 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42862: BlockingRecv error: recv got EOF from 127.0.0.1:42862 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260924 00:13:56.538511 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.537824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.61:43737)
0924 00:13:56.537914 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.537917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.537965 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:13:56.538042 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.538043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.538054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.538104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.538444 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42872: BlockingRecv error: recv got EOF from 127.0.0.1:42872 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 00:13:56.668663 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.667881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.98.61:43737)
0924 00:13:56.667968 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.667971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.668161 (+ 190us) server_negotiation.cc:407] Connection header received
0924 00:13:56.668191 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.668192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.668202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.668268 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.668578 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42884: BlockingRecv error: recv got EOF from 127.0.0.1:42884 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 00:13:56.926684 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:13:56.925940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.0.98.61:43737)
0924 00:13:56.926025 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:56.926027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:56.926041 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:13:56.926113 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:56.926114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:56.926125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:56.926180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:13:56.926589 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42888: BlockingRecv error: recv got EOF from 127.0.0.1:42888 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:13:57.163127 23197 catalog_manager.cc:1259] Loaded cluster ID: e2133d2829df427e8419317697af2a0f
I20260924 00:13:57.163252 23197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: loading cluster ID for follower catalog manager: success
I20260924 00:13:57.164125 23197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: acquiring CA information for follower catalog manager: success
I20260924 00:13:57.164487 23197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3bf42e4ba6c48ee90575ce3a66421f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:13:57.239439 23245 catalog_manager.cc:1259] Loaded cluster ID: e2133d2829df427e8419317697af2a0f
I20260924 00:13:57.239526 23245 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: loading cluster ID for follower catalog manager: success
I20260924 00:13:57.240499 23245 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: acquiring CA information for follower catalog manager: success
I20260924 00:13:57.240869 23245 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a27d782c166411584909097660e7f84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:13:57.442469 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:57.439891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.0.98.61:43737)
0924 00:13:57.440037 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:57.440039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:57.440051 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:13:57.440115 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:57.440116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:57.440191 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:57.440277 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:57.440486 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.440986 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.441753 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.441943 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.441997 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:57.442160 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:57.442283 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:57.442349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:57.442383 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 00:13:57.442401 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260924 00:13:57.446597 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:57.444341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.98.60:33005)
0924 00:13:57.444473 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:57.444476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:57.444484 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:13:57.444547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:57.444547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:57.444581 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:57.444648 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:57.444751 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.445239 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.445917 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.446096 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.446145 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:57.446277 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:57.446368 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:57.446424 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:57.446462 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 00:13:57.446477 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260924 00:13:57.451534 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:57.448517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.98.60:33005)
0924 00:13:57.448558 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:57.448559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:57.448568 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:13:57.448590 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:57.448591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:57.448601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:57.448657 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:57.448816 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.449600 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.450785 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.451014 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.451103 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:57.451263 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:57.451295 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:57.451364 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:57.451425 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 00:13:57.451444 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:13:57.451774 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:57.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.0.98.61:43737)
0924 00:13:57.448568 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:57.448570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:57.448950 (+ 380us) server_negotiation.cc:407] Connection header received
0924 00:13:57.448982 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:57.448982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:57.448994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:57.449057 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:57.449158 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.449853 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.450875 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.451181 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.451309 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:57.451478 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:57.451509 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:57.451535 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:57.451665 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 00:13:57.451702 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 00:13:57.452679 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:13:57.448314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.0.98.62:42237)
0924 00:13:57.448579 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 00:13:57.448581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:13:57.448588 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:13:57.448654 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:13:57.448654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:13:57.448705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:13:57.448791 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:13:57.449830 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.450616 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.451696 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:13:57.452024 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:13:57.452114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:13:57.452355 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:13:57.452448 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:13:57.452520 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:13:57.452574 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:13:57.452596 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":63,"threads_started":1}
I20260924 00:13:57.465807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 22962
I20260924 00:13:57.471522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 23037
I20260924 00:13:57.476702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskElJ0hL/build/release/bin/kudu with pid 23112
2026-09-24T00:13:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2964 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5794 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269288 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST